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Abstract 

Norden  (1970)  uses  the  Rayleigh,  which  is  a  degenerative  of  the  Weibull,  to 
model  manpower  on  researeh  and  development  (R&D)  programs.  Several  researeh 
efforts  extend  his  work  ineluding  Lee,  Hogue,  and  Gallagher  (1997)  who  build  R&D 
program  budgets  based  on  Rayleigh  expenditures.  We  demonstrate  the  theoretical 
limitations  to  the  Rayleigh  model  and  present  the  Weibull  model,  which  mitigates  those 
limitations.  Using  102  eompleted  R&D  defense  programs,  we  develop  regression  models 
to  prediet  the  requisite  shape  and  scale  parameters  to  forecast  Weibull-based  budgets. 
Using  the  remaining  26  eompleted  R&D  programs  to  validate  the  robustness  of  our 
regression  models,  we  show  that  100  and  96  percent  of  the  least  squares  estimated  shape 
and  seale  values  respeetively,  fall  within  a  95  percent  prediction  interval.  We  determine 
the  Weibull  model’s  budget  projection  capability  by  comparing  forecasted  Weibull-based 
budgets  to  128  eompleted  R&D  program  budgets  and  report  an  average  correlation  of 
0.607.  To  determine  the  significance  of  our  results  we  compare  forecasted  Rayleigh- 
based  budgets  to  the  same  128  completed  program  budgets.  Using  the  Weibull  over  the 
Rayleigh  model  when  applying  Lee,  Hogue,  and  Gallagher’s  (1997)  methodology,  we 
improve  initial  budget  profile  projections  on  average  60  percent. 
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FORECASTING  RESEARCH  &  DEVELOPMENT  PROGRAM  BUDGETS  USING 


THE  WEIBULL  MODEL 


I.  Introduction  to  the  Research 


Problem  Statement 

When  Military  Departments  plan  Research  and  Development  (R&D)  programs, 
they  must  project  a  budget  profile.  Lee,  Hogue,  and  Gallagher  (1997)  present  a  method 
to  derive  an  appropriate  budget  for  a  given  expenditure  profile.  Porter  (2001)  and  Unger 
(2001)  indicate  that  a  Weibull  distribution  model  fits  R&D  program  expenditures  well. 
When  a  program  begins;  however,  no  method  currently  exists  to  determine  the 
appropriate  Weibull  shape  and  scale  parameters.  This  research  provides  a  means  to 
determine  initial  R&D  program  budgets  based  on  a  Weibull  model. 

General  Issue 

The  program  manager  must  project  a  budget  profile  when  a  new  R&D  program  is 
planned.  However,  past  cost  and  schedule  projections  for  most  new  start  R&D  programs 
significantly  underestimate  the  final  program  cost  and  schedule.  Past  research  shows  that 
R&D  programs  historically  require  significant  increases  to  projected  costs  and  program 
schedule  duration.  Unger  (2001)  studied  37  historical  defense  R&D  programs  and  found 
that  the  final  budgets  and  actual  program  durations  exceed  the  initial  budgets  and 
estimated  program  durations  on  an  average  of  20  and  25  percent  respectively.  In  some 
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cases  the  program  cost  and  schedule  growth  is  very  extensive.  For  example,  Gallagher 
and  Lee  (1996)  report  that  the  R&D  effort  for  the  NavStar  Global  System  cost  nearly 
three  times  the  original  estimate,  and  the  Trident  Submarine  program  schedule  stretched 
from  5  to  16  years,  an  increase  of  320  percent.  The  Military  Departments  and  the  Office 
of  the  Secretary  of  Defense  (OSD)  Program  Analysis  and  Evaluation  (PA&E)  need  an 


accurate  way  to  determine  the  reasonableness  of  new  start  R&D  budget  projections. 


Figure  1.  Factors  Contributing  to  Development  Cost  (Belcher  and  Dukovich,  1999) 


Initial  program  development  costs,  schedules,  and  budget  profiles  are  difficult  to 
project  considering  the  number  of  possible  factors  that  affect  the  estimates.  Based  on 
expert  opinion,  Belcher  and  Dukovich  (1999)  present  a  macro  view  of  proposed 
contributors  to  how  development  programs  incur  costs.  Figure  1  shows  their  3  major  and 
12  supporting  contributors  they  propose.  Unger’s  (2001)  study  shows  that  1  of  the  12 
factors  (funding  constraints)  explains  53.4  percent  of  cost  overrun  and  50.5  percent  of 
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schedule  slip  variation.  Our  study  focuses  on  developing  a  method  for  determining  an 
appropriate  budget  for  new  R&D  program  starts  that  should  reduce  cost  and  schedule 
growth. 

Lee,  Hogue,  and  Gallagher  (1997)  present  a  method  to  derive  appropriate  budgets 
by  using  a  Rayleigh  distribution  to  project  an  expenditure  profile.  Unger  (2001) 
evaluates  37  final  R&D  program  expenditure  profiles  and  finds  that  only  52  percent  fit  a 
Rayleigh  distribution.  Porter  (2001)  and  Unger  (2001)  show  that  R&D  program  budgets 
more  often  support  a  Weibull  distribution  of  expenditures.  The  Rayleigh  distribution  is  a 
special  case  of  the  Weibull  distribution.  Since  the  Weibull  is  more  flexible  than  the 
Rayleigh,  we  expect  that  a  higher  percentage  of  expenditure  profiles  from  R&D  programs 
fit  a  Weibull  distribution. 

Research  Approach 

Lee,  Hogue,  and  Gallagher  (1997)  present  a  least  squares  approach  that  accounts 
for  spend  out  patterns  to  develop  annual  budgets  from  a  point  estimate.  They  use  the 
Rayleigh  distribution  with  predicted  total  program  cost  and  desired  duration  to  model 
expenditures.  Because  the  Rayleigh  distribution  suffers  from  theoretical  limitations,  we 
model  R&D  program  expenditures  with  a  Weibull  distribution  to  mitigate  those 
limitations.  However,  no  method  currently  exists  for  determining  the  Weibull  shape  and 
scale  parameters  at  program  inception.  Using  multiple  regression,  our  research 
determines  a  mathematical  relationship  between  the  Weibull  shape  and  scale  parameters 
with  predictors  including  lead  service,  type  of  program,  program  final  cost  and  duration. 
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While  we  use  actual  final  cost  and  duration  to  develop  this  approach,  program  managers 
will  have  to  rely  on  estimates. 

The  model  building  data  for  the  Weibull  shape  and  scale  parameters  are 
determined  by  fitting  a  Weibull  distribution  to  derived  program  expenditures.  The 
expenditure  profiles  are  derived  from  128  completed  R&D  program  budgets  from  the 
Selected  Acquisition  Report  (SAR)  by  applying  average  service  spend  out  rates.  We  test 
our  assumption  that  R&D  program  expenditures  fit  a  Weibull  model  using  Komolgorov- 
Smimov,  Cramer-von  Mises,  and  Anderson-Darling  goodness-of-fit  statistics.  The  study 
finds  a  relationship  between  the  Weibull  shape  and  scale  parameters  and  predictors  from 
the  SAR.  Our  method  predicts  the  requisite  shape  and  scale  parameters  for  the  Weibull 
model  to  provide  a  better  quantitative  method  to  determine  initial  R&D  program  budget 
profiles. 

Scope 

Several  factors  contribute  to  R&D  program  cost  overrun  and  schedule  slip.  This 
research  narrows  the  scope  of  contributing  factors  to  funding  constraints  attributed  to 
budget  profiles  that  inadequately  meet  fiscal  expenditure  requirements.  We  seek  to 
minimize  R&D  program  cost  and  schedule  growth  by  developing  a  better  quantitative 
approach  to  forecast  initial  program  budget  profiles.  Past  research  indicates  that  R&D 
budgets  result  in  expenditures  that  follow  a  Weibull  distribution.  This  research  seeks  to 
find  relationships  between  estimated  Weibull  parameters  for  completed  programs  and 
explanatory  variables  including  service  type,  program  type,  program  final  cost  and 
duration.  These  relationships  may  be  used  to  determine  an  expenditure  distribution  based 
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on  estimated  total  cost  and  desired  program  duration.  An  analyst  may  determine  a  budget 
from  Weibull-based  expenditures  using  Lee,  Hogue,  and  Gallagher’s  (1997)  approach. 

Research  Benefits 

The  research  seeks  to  minimize  R&D  program  cost  and  schedule  growth  by 
developing  a  better  quantitative  approach  to  project  initial  program  budget  profiles.  This 
quantitative  technique  is  potentially  useful  to  all  Military  Departments  and  OSD  PA&E 
to  verify  the  reasonableness  of  proposed  R&D  program  budget  profiles. 

Chapter  Summary 

This  chapter  proposes  the  Weibull  model  as  a  better  approach  to  forecasting  R&D 
program  budget  profiles.  Chapter  Two  explores  prior  research  that  uses  the  Rayleigh  and 
Weibull  models.  We  diseuss  the  limitations  to  the  Rayleigh  model  and  present  the 
Weibull  model,  whieh  mitigates  those  limitations.  Chapter  Three  explains  the 
methodology  applied  to  historieal  DoD  program  data  to  predict  the  requisite  shape  and 
scale  parameters  to  forecast  Weibull-based  R&D  budgets.  Chapter  Four  provides 
validation  results  to  the  methodology  applied  to  historical  DoD  program  data  outlined  in 
Chapter  Three.  We  include  an  average  correlation  comparison  between  Rayleigh-based 
and  Weibull-based  budget  profde  projections  to  128  completed  R&D  budget  profiles. 
Chapter  Five  summarizes  and  concludes  our  proposed  methodology,  while  addressing 
possible  limitations  and  future  research. 
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II.  Literature  Review 


Chapter  Overview 

Developing  a  budget  profile  that  meets  R&D  expenditure  requirements  is 
essential  to  the  success  of  a  new  program.  As  stated  in  Chapter  One,  Unger  (2001)  finds 
that  over  50  percent  of  cost  oveiTuns  and  schedule  slips  are  due  to  funding  constraints. 
Insufficient  program  funding  hinders  progress,  while  over-funded  programs  inefficiently 
use  resources  that  could  be  used  elsewhere.  Several  mathematical  models  are  employed 
to  forecast  an  appropriate  budget  profile.  Porter  (2001)  and  Unger  (2001)  review  the 
Beta,  Sech-Squared,  and  Rayleigh  Models.  Currently,  we  know  of  no  method  to  estimate 
the  parameters  at  program  inception  for  the  Beta  or  Sech-Squared  Models. 

This  chapter  relates  budgets  to  expenditures,  presents  Norden’s  theory  on 
Rayleigh  and  summarizes  several  applications.  We  present  the  Rayleigh  model,  identify 
Rayleigh  model  theoretical  limitations,  and  discuss  Porter  (2001)  and  Unger’s  (2001) 
findings  that  the  Weibull  model  better  fits  R&D  program  expenditures.  We  apply 
Norden’s  theory  in  terms  of  the  Weibull  function  and  present  and  graph  the  Weibull 
cumulative  distribution  function  and  probability  density  function.  This  chapter  concludes 
with  a  discussion  of  the  necessary  shape  and  scale  parameters  to  forecast  Weibull-based 
budget  profiles. 

When  Military  Departments  plan  R&D  programs,  they  must  project  a  budget 
profile.  The  defense  R&D  budget  for  any  given  year  expends  or  outlays  over  several 
years.  For  the  program  to  succeed,  the  budget  profile  must  meet  each  fiscal  year 
expenditure  requirement.  The  challenge  is  to  ensure  sufficient  funding  in  a  particular 
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year  given  each  budget  year’s  outlay  pattern.  Several  models  aid  in  forecasting  fiscal 
requirements,  we  present  the  Rayleigh  model  with  its  limitations  and  the  Weibull  model, 
which  mitigates  those  limitations.  Both  models  provide  an  approach  for  modeling  R&D 
program  expenditures  in  constant  dollars  over  time.  Each  of  the  models  estimates 
expenditures  by  applying  a  cost  factor  in  constant-year  dollars  to  their  respective 
cumulative  distribution  functions.  Therefore,  we  present  an  initial  discussion  of 
converting  budget  profiles  to  expenditure  profiles. 

Relating  Budget  Profiles  to  Expenditure  Profiles 

When  DoD  budgets  are  determined,  multi-year  appropriations  and  inflation  affect 
the  computation.  The  budget  or  total  obligation  authority  is  the  necessary  funding  to 
execute  the  program  over  multiple  years.  The  OSD  comptroller  prescribes  standard 
expenditure  patterns  or  percentages  of  R&D  funds  to  be  spent  in  a  given  year.  Outlay 
rates  are  the  percent  spent  for  each  particular  budget  year.  Inflation  indices  provide  the 
necessary  factor  to  calculate  budget  requirements  in  the  out  years  of  R&D  programs.  We 
convert  a  budget  profile  in  current  dollars  to  an  expenditure  profile  in  constant  dollars  by 
applying  the  outlay  rates  and  removing  inflation. 

We  first  apply  the  outlay  rates  to  the  budget  profile  to  obtain  the  expenditure 
profile,  Oi  in  current  dollars  for  each  rth  year  with 

Oi  =  BjS  I  +  Bi.]S2  +  Bi.2S3  + . . .  Bi.jsj,  ( 1 ) 

where  B,  is  the  budget  authority  for  the  /th  year  in  current-year  dollars,  sj  are  the  outlay 
rates  for  the yth  year  of  the  budget,  where  the  sum  of  1,  and  J  is  the  total  number  of 
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years  in  the  expenditure  profile.  The  result  is  the  expenditure  profile  in  current-year 
dollars,  which  are  yearly  expenditures  with  the  affects  of  inflation. 

We  then  convert  the  expenditure  profile  in  current-year  dollars  to  constant-year 
dollars  by  removing  the  inflation  component  with 

o,  =—  (2) 

c, 

where  O,.  are  the  expenditures  for  the  /th  year  in  constant-year  dollars,  O,  are  the 

expenditures  for  the  /th  year  in  current-year  dollars,  and  C/  is  the  inflation  index  for  the 
appropriate  /th  year.  The  current  dollar  budget  profile  is  now  an  expenditure  profile  in 
constant-year  dollars  using  formulas  (1)  and  (2).  The  expenditure  profile  in  constant-year 
dollars  is  required  to  utilize  the  Rayleigh  and  Weibull  models  that  theoretically  account 
for  development  effort,  whieh  is  not  affeeted  by  inflation.  The  next  section  presents 
Norden’s  theory  on  Rayleigh,  summarizes  numerous  applications  and  presents  and 
graphs  the  Rayleigh  model. 

Norden’s  Theory  on  Rayleigh 

Norden  (1970)  applies  the  Rayleigh  distribution  to  manpower  buildup  and  phase¬ 
out  on  development  efforts.  Norden  bases  his  theory  on  skill  level  increasing  linear  with 
time.  Because  skill  level  is  increasing  linear  with  respect  to  time,  the  rate  of  learning  is 
constant.  The  rate  at  which  R&D  programs  acquire  skills  is  not  affected  by  inflation, 
thus  the  Rayleigh  distribution  models  constant-year  expenditures.  The  initial  ramp-up  in 
the  Rayleigh  model  is  due  to  linear  skills  acquisition.  Since  the  rate  of  work  completed  is 
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proportional  to  the  work  remaining,  an  exhaustion  of  the  work  causes  the  exponential 
decrease  in  the  Rayleigh  tail  (Jarvis  and  Pohl,  1999:13). 

Putnam  (1978)  applies  the  Rayleigh  model  to  software  development.  Based  on 
budgetary  data  for  50  Computer  System  Command  systems,  Putnam  determines  that  the 
Rayleigh  estimates  software  development  well.  Since  many  software  programs  tend  to 
follow  the  characteristic  growth  to  a  peak  and  exponential  fall-off  of  the  Rayleigh, 
Putnam  declares  that  the  Raleigh  model  relates  to  software  development. 

Later  research  broadens  the  Rayleigh  model  application  to  DoD  contracts. 

Watkins  (1982)  applies  the  Rayleigh  model  to  defense  acquisition  cost  and  schedule  data 
on  30  DoD  contracts.  Watkins  reports  that  the  Rayleigh  modeled  the  earned  value  of 
contracts  well.  Abemethy  (1984)  applies  the  Rayleigh  model  to  21  completed  Navy 
contracts.  Although  Abemethy  did  not  meet  his  objective  to  use  the  Rayleigh  model  as  a 
forecasting  tool,  he  finds  the  Rayleigh  adequately  modeled  DoD  contract  data.  Lee, 
Hogue,  and  Gallagher  (1997)  continue  the  application  by  demonstrating  that  the  Rayleigh 
model  fits  defense  R&D  acquisition  program  expenditures  in  constant-year  dollars.  Past 
research  substantiates  that  the  Rayleigh  CDF  models  cumulative  constant-dollar  R&D 
defense  program  expenditures  well  (Gallagher  and  Lee,  1996:52). 

Norden’s  Theory  in  Terms  of  the  Weibull 

The  Rayleigh  model  is  prevalent  in  R&D  expenditure  modeling  (Norden,  1970; 
Putnam,  1978;  Lee,  1997;  Gallagher,  1996).  Norden  (1970)  develops  the  Rayleigh  model 
based  upon  engineers  solving  a  fixed  number  of  development  project  problems  with 
increased  linear  learning.  Although  he  states  linear,  Norden  notes  experiments  with  other 
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than  linear  rates,  which  leads  to  the  general  class  of  the  Weibull  models.  Porter  (2001) 
and  Unger  (2001)  show  that  the  Weibull  function  models  R&D  program  expenditures  at  a 
higher  degree  of  accuracy  than  the  Rayleigh  model. 

The  Weibull  and  Rayleigh  are  Related.  The  Rayleigh  function  is  a  degenerative 
of  the  Weibull  function.  Hines  and  Montgomery  (1980)  present  the  Weibull  cumulative 
distribution  function  (CDF)  with  the  location  parameter  as 

-f— f 

(3) 

and  the  Weibull  probability  density  function  (PDF),  the  derivative  of  the  Weibull  CDF, 
as 


/(0=| 


^  t-y'^ 


(4) 


where  t  is  time  in  years,  y  is  the  location  parameter,  S  is  the  scale  parameter,  and  {5  is 
the  shape  parameter.  The  results  of  the  Weibull  CDF  with  (3)  always  fall  between  0  and 
1.  The  cost  factor  d  scales  the  Weibull  CDF  to  reflect  the  cumulative  program 
expenditures.  We  present  the  Weibull  cost  model  as  the  CDF  in  (3)  multiplied  by  a  cost 
factor  d  with 


E{t)  =  d 


\-e 


t-r 


(5) 


V  J 

The  Weibull  cost  model  calculates  a  program’s  total  cumulative  expenditures  at  a 
specified  time  t,  for  a  given  shape,  scale,  and  location  parameter.  The  Rayleigh  CDF  is  a 
special  case  of  the  Weibull  CDF  in  (3)  when  fixing  the  shape  and  location  parameters  to 
13=  2  and  y=  0.  We  define  the  Rayleigh  CDF  as 
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(6) 


F{t) 


and  the  Rayleigh  PDF  as 

f{t)  =  2S-^te-^"‘\  (7) 

The  foundational  concept  for  the  Weibull,  and  hence  the  Rayleigh,  cost  models  is 
based  upon  solving  a  fixed  number  of  development  project  problems  with  linear  learning. 
The  rate  at  which  work  is  completed  is  a  function  of  performance  and  remaining  work, 

dw{t) 


dt 


p(t)[\-w(t)]. 


where  p(t)  is  the  performance  and  w(t)  is  the  remaining  work.  Let  z{f)  =  1-  w{f),  so 
dz{t)  -dwit)  dz{t) 


dt 


dt 


•and 


dt 


■  =  -p{t)z{t)  we  solve  for  w{t).  By  integrating 


(.=■1 

ln(z(t))  =  -  jp(r)dr  and  evaluating  both  sides  to  the  power  of  the  base  of  the  natural 

r=0 


-  fp(r)c/r 

logarithm,  we  obtain  z(t)  -e  .  We  show  the  total  work  completed  by  substituting 

back  in  percent  of  work  remaining  at  time  t,  w{t),  in  the  following  form 

r=t 

-  \p(r)d{T:) 

w{t)  =  1  —  e 

We  define  performance  on  the  program  for  any  given  time  as  a  constant  k  multiplied  by 


time  to  a  constant  power,  p(t)  =  ki’.  Since  f  p{T)dT  =  f  kz^’dr  = - 

^  *'  b  +  \ 

T=0  r=0  u-ri 


k  .b+\ 


■t 

^  9 


w{t)  =  \-e 


6+1 
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We  obtain  the  Weibull  CDF  in  (3)  when  b+\  and  d=’’*-^b  +  \l k  .  We  obtain  the 
Rayleigh  CDF  with  linear  growth  in  performance  over  time  when  the  location  parameter 
Y=  0,  /3=  2,  and  5=^2lk  .  The  percent  work  complete  according  to  the  Weibull  CDF  is 
derived  if  performance  improves  over  time  to  a  power  other  than  one. 


The  Rayleigh  CDF.  Figure  2  presents  the  Rayleigh  CDF.  We  apply  formula  (6) 
given  S=  6  and  P=2.  This  figure  shows  the  cumulative  percent  of  expenditures  incurred 
for  an  R&D  effort. 


Figure  2.  Rayleigh  Cumulative  Distribution  Function  (CDF) 


The  Rayleigh  PDF.  The  derivative  of  (6)  gives  the  Rayleigh  PDF  in  (7).  Figure  3 
shows  the  Rayleigh  PDF  with  the  same  parameters  as  in  Figure  2.  The  Rayleigh  PDF  is 
the  instantaneous  rate  that  the  percentage  funds  are  expended. 
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Figure  3.  Rayleigh  Probability  Density  Function  (PDF) 


The  Rayleigh  Model  Parameters.  The  two  varying  parameters  in  the  Rayleigh 
funetion  are  the  seale  and  eost  faetor.  The  scale  parameter,  S,  determines  the  steepness  of 
the  Rayleigh  CDF  curve,  and  the  cost  factor,  d,  scales  the  Rayleigh  CDF  to  reflect  the 
cumulative  program  expenditures.  The  S  parameter  determines  the  time  period  that 
manpower  utilization  is  maximized  (Norden,  1970:126).  The  results  of  the  Rayleigh 
CDF  with  (6)  always  fall  between  0  and  1.  We  model  the  cumulative  expenditures  with 


E{t)^d 


\-e 


(8) 


where  d  is  the  cost  factor  of  the  R&D  project  and  t  is  the  time  from  program  inception  to 
completion  (Lee,  Hogue,  and  Gallagher,  1997:31).  Figure  4  shows  how  we  apply  (8)  to 
illustrate  the  increasing  gradient  of  the  Rayleigh  CDF  scaled  with  d  as  the  d  parameter 
decreases. 
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Figure  4,  The  Rayleigh  CDF  while  changing  the  5  parameter  with  d  -  1000 


The  derivative  of  the  Rayleigh  cost  model  (8)  is  the  scaled  Rayleigh  PDF  or 
expenditure  rate  for  a  modeled  program.  We  define  the  Rayleigh  PDF  scaled  with  the  d 
parameter  as 

(9) 

Figure  5  demonstrates  the  effects  of  changing  the  S  parameter  on  the  Rayleigh  PDF. 
Notice  that  as  the  S  parameter  decreases,  the  time  of  peak  expenditures  occurs  earlier  and 
higher  in  the  program.  The  maturity  of  a  program  is  sooner  when  the  rate  of  expenditures 
is  higher  than  another  program  of  equal  value  (in  constant  dollars). 
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Figure  5.  The  Rayleigh  PDF  while  changing  the  5  parameter  with  d  -  1000 


Figure  5  illustrates  how  the  Rayleigh  expenditure  distribution  provides  important 
information  about  a  modeled  program.  The  shape  of  the  expenditure  distribution 
determines  the  peak  time  and  magnitude  of  expenditures  and  the  estimated  program 
duration.  Lee,  Hogue,  and  Gallagher  (1997)  present  two  methods  to  determine  the 
appropriate  d  parameter.  The  first  technique  is  based  on  the  time  of  peak  expenditure 
rate  and  the  other  is  based  on  the  estimated  program  completion  time. 

The  Time  of  Peak  Expenditure  Rate  Technique.  This  technique  uses  a  program’s 
estimated  time  of  peak  expenditure  rate  to  determine  the  5  parameter.  The  time  of  peak 
expenditure  rate  is  estimated  with  some  degree  of  reliability.  For  instance,  aircraft  R&D 
program’s  peak  expenditure  rate  typically  occurs  at  the  time  of  first  test  flight  (Lee, 
Hogue,  and  Gallagher,  1997:32).  If  the  peak  expenditure  rate  is  known  then  the  d 
parameter  is  defined  as 

=  CO) 
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by  setting  (7)  equal  to  zero  (Lee,  Hogue,  and  Gallagher,  1997:32).  Setting  (7)  equal  to  0 
and  solving  for  t  results  in  the  peak  of  a  program’s  expenditure  rate  with  respect  to  time. 

The  Estimated  Program  Completion  Time.  When  the  time  of  peak  expenditure 
rate  is  unknown,  we  use  the  estimated  program  completion  time  to  determine  an 
appropriate  d  parameter.  Because  the  right  side  of  the  Rayleigh  distribution  has  an 
infinite  tail,  a  defined  point  must  determine  program  completion.  Lee,  Hogue,  and 
Gallagher  (1997)  define  this  point  as  tfmai,  the  time  of  final  development.  Final 
development  occurs  when  the  Rayleigh  CDF  reaches  97  percent  of  the  constant-dollar 
cumulative  R&D  expenditures  for  a  project  in  the  Engineering,  Manufacturing,  and 
Development  phase  according  to  Lee,  Hogue,  and  Gallagher  (1997).  Lee,  Hogue,  and 
Gallagher  define  the  equation  as 

D=E(tflnal)=0.97d,  (11) 

where  D  is  the  total  R&D  estimated  program  eost  in  eonstant  dollars  and  d  is  the 
parameter  that  seales  the  Rayleigh  CDF  to  program  eost.  When  the  program  completion 
time  is  estimated,  then  Lee,  Hogue,  and  Gallagher  (1997)  define  the  d  parameter  as 

<S  =  0.5345/^„.,.  (12) 

In  conclusion,  either  the  time  of  peak  expenditure  rate  or  the  estimated  program 
completion  time  technique  determines  an  appropriate  d  parameter  for  the  Rayleigh 
model.  The  time  scale  parameter,  d,  and  cost  factor,  d,  provide  the  necessary  information 
to  utilize  the  Rayleigh  model.  Lee,  Hogue,  and  Gallagher  (1997)  demonstrate  how  a 
budget  profile  forecast  is  determined  given  an  initial  total  R&D  program  cost  estimate,  D, 
and  either  the  time  of  peak  expenditures,  tpeak,  or  the  estimated  program  completion  time. 
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tfinai-  We  turn  our  attention  now  to  the  characteristics  that  limit  the  Rayleigh  model  in  this 
application. 

The  Limitations  of  the  Rayleigh  Model.  There  are  two  characteristics  that 
theoretically  limit  the  Rayleigh  distribution  in  modeling  R&D  program  expenditures. 

The  Weibull  distribution  simplifies  to  the  Rayleigh  distribution  when  removing  the 
location  parameter,  y,  and  fixing  the  shape  parameter  fi=2.  These  two  characteristics  of 
the  Rayleigh  distribution  make  the  Rayleigh  model  somewhat  rigid  in  its  ability  to  model 
program  expenditures. 

The  Rayleigh  Shape  Parameter  B=  2.  This  characteristic  of  the  Rayleigh 
distribution  results  in  the  peak  expenditures  oeeurring  approximately  at  the  38  percentile 
of  the  total  program  duration  (Gallagher  and  Lee,  1996:52).  However,  a  program 
expenditure  rate  may  peak  earlier  or  later.  For  example,  a  program  might  have 
expenditures  that  stop  shortly  after  the  time  of  peak  rate  of  expenditures,  indicating  a 
very  short  tail.  In  this  ease  the  Rayleigh  shape  parameter,  which  is  fixed  at  two,  causes 
the  Rayleigh  distribution  to  foreeast  reality  inaeeurately.  The  solution  to  this  problem  is 
allowing  the  shape  parameter  to  vary,  whieh  leads  us  to  the  Weibull  distribution. 

The  Rayleigh  Distribution  Cannot  Model  Insignificant  Funding.  The  Rayleigh 
model  does  not  have  a  location  parameter,  whieh  gives  the  Weibull  distribution  flexibility 
in  modeling  the  relative  start  of  a  program.  For  example,  initial  R&D  programs 
sometimes  start  out  with  a  few  years  of  minimal  funding.  The  Rayleigh  distribution  lacks 
the  ability  to  model  this  insignificant  funding.  The  location  parameter  allows  the  Weibull 
distribution  to  model  programs  with  no  funding  for  the  initial  years  of  insignificant 
funding. 
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In  summary,  the  Rayleigh  function  is  somewhat  rigid  in  its  ability  to  model  R&D 
expenditures.  The  varying  shape  parameter  and  additional  location  parameter  make  the 
Weibull  distribution  more  flexible  in  modeling  R&D  program  expenditures.  For  this 
reason  our  discussion  now  focuses  on  the  Weibull  function. 

The  Weibull  Model’s  Flexibility 

When  discussing  the  Weibull  verses  the  Rayleigh  model  there  are  two  major 
differences.  Notice  that  (3)  and  (6)  are  equivalent  with  the  exception  of  the  location 
parameter  and  the  fixed  shape  parameter  y^=  2.  The  location  parameter  gives  flexibility 
to  the  Weibull  function  to  model  program  expenditures  when  relatively  insignificant 
funding  occurs  at  the  beginning  of  the  program.  Because  the  Weibull  shape  parameter 
varies,  the  time  of  peak  expenditures  does  not  fix  the  program  completion  time.  Since  a 
fixed  shape  parameter  does  not  accurately  model  R&D  program  expenditures  as  well  as 
one  that  varies,  we  use  the  three-parameter  Weibull  function  allowing  the  shape 
parameter  to  vary.  We  obtain  a  better  fit  of  the  data  using  the  Weibull  model  as  opposed 
to  the  Rayleigh  model. 

The  Weibull  function  provides  the  flexibility  to  estimate  various  distributions. 

For  example,  when  the  shape  parameter,  (5=  1,  it  produces  an  exponential  distribution. 
When  15=  2,  as  stated  earlier,  it  produces  the  Rayleigh  distribution.  For  15=  3.4,  the 
Weibull  is  approximately  normally  distributed.  The  Weibull  shape  parameter  determines 
the  time  of  peak  expenditures.  Figure  6  shows  the  effects  of  changing  the  Weibull  shape 
parameter  when  holding  the  location  and  scale  parameters  constant. 
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Figure  6.  Effects  of  changing  Weibull  shape  with  y-Q,  5-'i,d-  1000 

Figure  7  demonstrates  the  effects  of  changing  the  Weibull  scale  d  parameter  while 
holding  the  shape  and  location  parameters  constant.  Increasing  the  scale  parameter 


Figure  7.  Effects  of  changing  Weibull  scale  d,  with  y-^i,  P-2,d-  1000 

Figure  8  illustrates  the  Weibull  location  parameter’s  ability  to  model  insignificant 
funding  in  the  first  few  years  of  the  program.  In  essence,  the  location  parameter  changes 
the  start  time  t  when  R&D  program  expenditures  are  significant. 
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Figure  8.  Effects  of  changing  Weibull  Location  %  with  fi-2,  S-3,d-  1000 


In  summary,  the  characteristics  of  the  Weibull  models  have  an  initial  ramp-up,  a 
peak,  and  an  exponential  tail.  The  initial  ramp  up  is  due  to  the  improvement  in 
performance.  The  exponential  decrease  in  the  tail  is  caused  by  the  exhaustion  of  the 
work,  since  work  performed  is  proportional  to  work  remaining. 

Converting  a  Point  Estimate  to  a  Budget  Profile 

When  Military  Departments  plan  Research  and  Development  (R&D)  programs, 
they  must  project  a  budget  profile.  The  budget  or  total  obligation  authority  (TOA)  is  the 
necessary  funding  to  execute  the  program  over  multiple  years.  The  OSD  comptroller 
prescribes  standard  expenditure  patterns  or  percentages  of  R&D  funds  to  be  spent  in  a 
given  year.  The  targeted  percentage  of  funds  spent  in  a  particular  budget  year  is  called  an 
outlay  rate.  Inflation  is  also  used  to  calculate  the  necessary  budget  requirements  in  the 
out  years  of  R&D  programs. 

The  success  of  a  program  is  determined  by  several  factors.  One  important  factor 
that  this  thesis  effort  hopes  to  improve  is  a  forecasted  budget  profile  that  will  better  meet 
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each  fiscal  year  expenditure  requirement.  The  challenge  is  to  ensure  sufficient  funding  in 
a  particular  year  given  each  budget  year’s  outlay  pattern. 

Lee,  Hogue,  and  Gallagher  (1997)  present  a  method  to  derive  the  appropriate 
budget  for  a  given  total  program  cost  estimate  using  Rayleigh  distributed  expenditures. 
This  thesis  effort  uses  their  method  of  forecasting  R&D  program  budgets,  but  with 
Weibull  distributed  expenditures.  We  present  an  example  of  Lee,  Hogue,  and 
Gallagher’s  (1997)  method  of  determining  a  budget  profile  using  the  Rayleigh  model 
when  given  R&D  program  estimates  for  total  cost,  D,  and  completion,  tfi„ai. 

We  demonstrate  a  simple  hypothetical  program  with  R&D  expenditures  of  500 
million  dollars  occurring  over  10  years.  The  program  is  in  base  year  2000.  Given  D  = 
500  and  tflnai=  10,  the  Rayleigh  model  parameters  ealeulated  with  (11)  and  (12)  are  d  = 
515.46  and  d=  5.345.  Table  1  eolumn  3  shows  the  Rayleigh  cumulative  constant-dollar 
expenditures,  ealeulated  using  (8).  Sinee  the  Rayleigh  cumulative  expenditures  must  be 
converted  to  annual  expenditures,  O.  for  the  rth  year  of  the  program,  we  use  the  formula 

d,=£(/,)-i<  (/,._,),  (13) 

where  E  is  the  constant-dollar  cumulative  expenditures  in  (8)  and  time  6  is  the  fiscal  year 
end  i.  The  hat  represents  modeled  values,  different  from  budget  derived  values  with  (1) 
and  (2).  Table  1  column  4  shows  the  annual  constant-dollar  expenditures  and  column  5 
shows  the  annual  current-year  dollars  by  multiplying  the  Air  Force  raw  inflation  indices, 
Ci,  shown  in  the  last  column. 
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Table  1.  Rayleigh-Based  Expenditure  Profile  Example 


Fiscal 

Year 

Time 

Rayleigh 

Cumulative 

Expenditures 

CY00$M 

Rayleigh 

Annual 

Expenditures 

CYOOSM 

Rayleigh 
Annual 
Expenditures 
Current  $M 

Air  Force 
Raw 
Inflation 
Indices 

2000 

1 

17.73 

17.73 

17.73 

■■■ 

2001 

2 

67.34 

49.61 

50.36 

mSSSM 

2002 

3 

139.28 

71.94 

74.12 

mSwM 

2003 

4 

221.03 

81.74 

85.48 

2004 

5 

300.59 

79.56 

84.86 

1.0666 

2005 

6 

369.25 

68.66 

74.70 

1.0879 

2006 

7 

422.70 

53.45 

59.31 

1.1097 

2007 

8 

460.59 

37.89 

42.89 

1.1319 

2008 

9 

485.20 

24.61 

28.41 

1.1545 

2009 

10 

499.90 

14.70 

17.31 

1.1776 

Total 

499.90 

535.16 

To  this  point  of  the  example,  several  key  points  are  made.  The  Rayleigh 
cumulative  expenditures  show  the  necessary  funding  for  the  program’s  success  in 
constant  dollars.  The  Rayleigh  annual  expenditures  in  constant  dollars  provide  the 
funding  needed  in  base  year  2000.  The  Rayleigh  annual  expenditures  in  current  dollars 
provide  the  funding  needed  in  the  future  years,  taking  into  account  inflation.  For 
example  in  Table  1,  in  FY2009,  the  program  needs  $14.7M,  expressed  in  FY2000  dollars 
or  equivalently  $17.3  IM  expressed  in  FY2009  dollars.  For  the  program  manager  to  meet 
program  expenditure  requirements,  the  sum  of  previous  and  cuiTent  budgets  multiplied  by 
the  outlay  rates  should  equal  the  annual  expenditures  in  cun'ent-year  dollars. 

The  developed  expenditure  profile  provides  how  much  the  program  will  spend 
each  year.  The  task  is  to  determine  a  budget  profile  that  will  meet  a  program’s 
expenditure  requirements.  There  is  a  difference  between  a  budget  profile  and  an 
expenditure  profile.  The  budget  profile  contains  the  necessary  funds  for  the  life  of  the 
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program  taking  into  account  the  OSD  Comptroller  outlay  rates  and  inflation.  The 
expenditure  profde  is  eomprised  of  a  mixture  of  different  years  of  budgeted  funds 
aeeording  to  the  outlay  pereentages. 

The  example  eontinues  with  our  approach  to  develop  the  neeessary  TOA  or 
budget  profile  to  meet  fiseal  expenditure  requirements.  To  develop  a  budget  profile  we 
must  take  into  aeeount  that  the  total  appropriation  for  an  R&D  program  eannot  be  spent 
during  the  year  in  whieh  it  is  authorized.  The  outlay  rates  govern  the  amount  of  yearly 
budget  funds  a  program  is  allotted  each  year. 

For  our  example  we  use  hypothetical  OSD  comptroller  published  outlay  rates  of 
0.540,  0.324,  0.090,  0.020,  0.01 1,  0.004,  and  0.002  for  the  first  through  seventh  year 
respeetively.  This  would  suggest  that  if  a  program  had  $10M  appropriated  to  it  in  the 
first  year,  only  $5.4]V1  should  be  expended.  The  second  year  the  program  manager  should 
expend  $3.24]V1  of  the  first  year’s  budget  authority.  If  the  expenditure  requirements  for 
the  second  year  of  the  program  are  $15M,  then  the  program  manager  should  ask  for 
$21.78M  in  appropriations  for  the  second  year.  The  example  calculation  is  provided 
($15M  needed  -  $3.24M  Yr-1  appropriation)/(.5400  Yr-1  outlay  rate)  =  $21.78M. 

The  complexity  of  developing  a  budget  profile  compounds  according  to  the 
duration  and  size  of  the  program.  The  expenditure  profile  is  easily  forecasted  with  the 
Rayleigh  and  Weibull  models.  The  challenge  is  converting  that  expenditure  profile  to  a 
budget  profile  when  taking  into  account  relevant  appropriation  years  related  to  different 
outlay  rates.  Lee,  Hogue,  and  Gallagher  (1997)  provide  a  nonlinear  estimation  approaeh, 
whieh  is  our  final  step  in  forecasting  an  R&D  program  budget  profile. 
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This  method  allows  the  budget  estimate,  for  each  budget  year  /,  to  change 


simultaneously  until  an  optimal  solution  is  reached.  We  apply  this  approach  by 
substituting  the  5,.  in  (1)  and  (2)  and  allow  Microsoft  Excel  Solver  function  (2000)  to 

select  the  yearly  budget  estimates  that  minimize  the  sum  of  squared  errors  between  the 
Rayleigh  modeled  expenditures  and  the  actual  expenditures  that  result  from  the 
forecasted  budget  profile.  Specifically, 

N+J-\ 

minX(5,-A)'  (14) 

;=1 

where  B.>Q  and  6.  is  the  Rayleigh  modeled  expenditure  profile  (13)  and  O.  is  the 

actual  expenditure  profile  for  each  /th  year  from  (1)  and  (2).  N+J-1  represents  the  total 
program  and  outlay  years  to  ealeulate.  Table  2  shows  the  results  of  our  example  program 
with  a  generated  budget  profile  in  the  final  eolumn. 


Table  2.  Rayleigh-Based  Budget  Profile  Projection  Example 


Fiscal 

Year 

Time 

Annual  Current  $M  Expenditures 

Error 

Budget  Profile 
Current  $M 

Estimated 

2000 

1 

17.73 

17.75 

-0.02 

0.00 

34.43 

2001 

2 

50.36 

50.32 

0.04 

0.00 

73.13 

2002 

3 

74.12 

74.18 

-0.06 

0.00 

87.14 

2003 

4 

85.48 

85.37 

0.11 

0.01 

91.72 

2004 

5 

84.86 

85.04 

-0.18 

0.03 

83.32 

2005 

6 

74.70 

74.41 

0.29 

0.09 

66.49 

2006 

7 

59.31 

59.81 

-0.50 

0.25 

50.61 

2007 

8 

42.89 

42.01 

0.88 

0.78 

29.97 

2008 

9 

28.41 

29.92 

-1.51 

2.27 

24.38 

2009 

10 

17.31 

14.33 

2.99 

8.92 

2.12 

2010 

11 

0.00 

3.94 

-3.94 

15.53 

0.00 

2011 

12 

0.00 

1.21 

-1.21 

1.46 

0.00 

2012 

13 

0.00 

0.35 

-0.35 

0.12 

0.00 

2013 

14 

0.00 

0.09 

-0.09 

0.01 

0.00 

2014 

15 

0.00 

0.01 

-0.01 

0.00 

0.00 

Total 

535.16 

538.72 

-3.56 

29.47 

543.30 
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Chapter  Summary 

Based  on  past  research,  the  Rayleigh  function  provides  a  model  to  determine 
R&D  expenditure  profiles.  This  chapter  details  these  fundamental  assumptions  starting 
with  Norden  who  used  the  Rayleigh  to  model  manpower  and  development  effort  over 
time.  Putnam  derived  cumulative  software  costs  using  the  Rayleigh  model,  and  Lee, 
Hogue,  and  Gallagher  employed  the  Rayleigh  to  model  defense  development 
expenditures.  We  justify  using  the  Weibull  function  in  modeling  expenditure  profiles 
according  to  the  generalizations  of  the  Rayleigh  model.  Following  the  description  of  the 
parameter  characteristics,  and  limitations  of  the  Rayleigh  function,  we  focus  on  the 
flexible  Weibull  function.  After  deriving  the  Weibull  function  we  explain  the  parameters 
and  discuss  its  ability  to  model  R&D  program  expenditures.  A  brief  discussion  of  Lee, 
Hogue,  Gallagher’s  method  of  foreeasting  a  budget  profile  from  an  R&D  cost  estimate 
eoneludes  this  ehapter. 
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III.  Research  Methodology 


Chapter  Overview 

Past  research  shows  that  the  Rayleigh  cumulative  distribution  function  models 
R&D  program  expenditures  well.  The  Weibull  function  models  R&D  program 
expenditures  better  than  the  Rayleigh.  This  research  employs  the  Weibull  function  to 
forecast  an  initial  R&D  budget  profile  by  developing  regression  models  to  predict  the 
necessary  Weibull  shape  and  scale  parameters.  A  description  of  the  methodology  to 
develop  the  predictive  shape  and  scale  regression  models  to  forecast  Weibull-based 
budgets  is  the  focus  of  this  chapter. 

Appropriate  data  collection  is  the  initial  step  in  this  research.  The  Selected 
Acquisition  Report  (SAR),  maintained  by  the  Office  of  the  Secretary  of  Defense  is  our 
source  of  data.  For  each  R&D  program  we  collect  various  categorical  characteristics  and 
the  final  annual  budget  profile  data.  After  we  convert  each  budget  profile  in  current 
dollars  to  expenditure  profiles  in  fiscal  year  2000  constant  dollars,  we  estimate  the 
Weibull  shape  and  scale  parameters.  We  test  the  assumption  that  the  Weibull  distribution 
fits  program  expenditures  using  Komolgorov-Smimov,  Cramer-von  Mises,  and 
Anderson-Darling  goodness-of-fit  (GOF)  statistics.  The  least  squares  estimated  shape 
and  scale  parameters  are  the  responses  or  dependent  variables  we  predict  in  our 
regression  models.  The  final  cost  and  schedule  with  various  categorical  data  like  branch 
of  military  service  and  type  of  program  provide  possible  predictors  or  independent 
variables  in  our  regression  models. 
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We  randomly  select  80  percent  of  the  full  data  set  to  build  our  Weibull  shape  and 
scale  regression  models,  and  set  aside  the  remaining  20  percent  for  validation.  We  test 
for  a  mathematical  relationship  between  the  Weibull  shape  and  scale  parameters  against 
possible  predictors  like  military  branch  of  service,  program  type,  total  cost  and  duration. 
Using  multiple  regression  we  develop  predictive  models  for  the  Weibull  shape  and  scale 
parameters.  To  validate  the  robustness  of  our  resulting  regression  models,  we  determine 
if  the  remaining  20  percent  completed  R&D  program  least  squares  estimated  shape  and 
scale  validation  data  values  fall  within  a  95  percent  prediction  interval.  To  determine  the 
Weibull  model’s  budget  forecasting  capability,  we  compare  forecasted  Weibull-based 
budgets  to  128  completed  R&D  program  budgets  using  Lee,  Hogue,  and  Gallagher’s 
(1997)  methodology.  Using  the  same  methodology  we  determine  the  significance  of  our 
results  by  eomparing  the  average  correlation  of  foreeasted  Weibull-based  and  Rayleigh- 
based  budgets  to  the  same  128  eompleted  program  budgets. 

Program  Data  Colleetion 

We  gather  R&D  program  funding  data  from  the  Selected  Acquisition  Reports 
(SARs)  to  evaluate  our  research  hypothesis.  Our  data  collection  comprises  final 
Research,  Development,  Test  and  Evaluation  program  annual  budget  profiles  to 
Milestone  111  or  equivalent,  military  branch  of  service,  program  type,  and  the  final 
program  cost  and  duration.  Since  the  SAR  did  not  report  annual  budget  data  prior  to 
1982,  our  data  collection  includes  only  those  programs  with  a  final  SAR  report  dated 
1982  or  later.  The  SAR  presents  the  budget  profiles  in  millions  of  both  constant  and 
current-year  dollars.  We  collect  only  current-year  budget  profiles  for  the  purpose  of 
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converting  current  to  constant-year  dollars  consistently  across  programs.  Our  selection 
criteria  include  programs  that  were  not  terminated  with  at  least  a  three-year  budget 
profile  to  Milestone  111.  Our  full  data  set  comprises  128  R&D  programs.  Table  3 
presents  the  Air  Force’s  Airborne  Warning  and  Control  System  (AWACS)  Radar  System 
Improvement  Program  (RSIP)  as  an  example  final  SAR  budget  profile.  We  graphically 
display  the  RSIP  final  program  budget  profile  in  Figure  9. 


Table  3.  RSIP  Final  Program  Budget  Profile 


Fiscal  Year 

1989 

1990 

1991 

1992 

1993 

1994 

1995 

1996 

Total 

Current  $M 

44.2 

63.7 

71.8 

117.1 

15.4 

38.4 

42.7 

31.1 

424.4 

u 

3 
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Figure  9.  RSIP  Final  Budget  Profile 


Convert  Budget  Profiles  to  Expenditure  Profiles 

We  convert  the  program  budget  data  into  constant-dollar  expenditures  in  two 
steps:  1)  convert  the  budget  profile  into  an  expenditure  profile  and  2)  adjust  for  inflation. 
The  first  step  is  to  convert  the  current-dollar  budget  profile  into  current-dollar 
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expenditures.  We  achieve  this  conversion  using  (1)  when  multiplying  each  budget  year 
by  the  appropriate  outlay  rates  and  summing  the  expended  funds  for  each  fiscal  year. 
Since  the  yearly  outlay  rates  vary  slightly  from  year  to  year,  we  use  average  outlay  rates 
from  1993  to  2001  for  our  calculations. 


Table  4,  Average  Service  R&D  Outlay  Rates  (as  Percentages) 


Air  Force 

FYOl 

FYOO 

FY99 

FY98 

FY97 

FY96 

FY95 

FY94 

FY93 

Avg 

StDev 

Yrl 

59.5 

58.8 

59.1 

50.7 

45.8 

46.3 

46.5 

46.5 

50.8 

51.56 

5.98 

Yr2 

33.7 

34.5 

33.1 

37.4 

39.9 

39.1 

38.8 

38.8 

34.5 

36.64 

2.67 

Yr3 

3.6 

3.6 

5.3 

6.8 

8.9 

8.9 

8.8 

8.9 

9.5 

7.14 

2.40 

Yr4 

1.0 

1.0 

1.4 

3.0 

3.6 

3.6 

3.6 

3.6 

3.4 

2.69 

1.19 

Yr5 

0.3 

0.3 

0.4 

0.8 

1.1 

1.1 

1.1 

1.1 

1.1 

0.81 

0.37 

Yr6 

0.0 

0.3 

0.3 

0.3 

0.3 

0.4 

0.4 

0.4 

0.4 

0.31 

0.13 

Yr7 

0.0 

0.0 

0.2 

0.0 

0.1 

0.0 

0.2 

0.0 

0.2 

0.08 

0.10 

98.1 

98.5 

99.8 

99.0 

99.7 

99.4 

99.4 

99.3 

99.9 

99.23 

Army 

FYOl 

FYOO 

FY99 

FY98 

FY97 

FY96 

FY95 

FY94 

FY93 

Avg 

StDev 

Yrl 

57.5 

56.8 

58.0 

58.0 

58.0 

57.0 

57.0 

55.0 

55.0 

56.92 

1.19 

Yr2 

32.5 

33.7 

33.0 

33.0 

33.0 

34.0 

34.0 

34.0 

34.0 

33.47 

0.59 

Yr3 

6.3 

5.0 

5.3 

5.3 

5.3 

5.3 

5.3 

7.3 

7.3 

5.82 

0.91 

Yr4 

2.1 

2.1 

2.1 

1.8 

1.8 

1.8 

1.8 

1.8 

1.8 

1.90 

0.15 

Yr5 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.8 

0.80 

0.00 

Yr6 

0.0 

0.5 

0.5 

0.5 

0.5 

0.5 

0.5 

0.5 

0.5 

0.44 

0.17 

Yr7 

0.0 

0.0 

0.2 

0.0 

0.2 

0.0 

0.2 

0.0 

0.2 

0.09 

0.11 

99.2 

98.9 

99.9 

99.4 

99.6 

99.4 

99.6 

99.4 

99.6 

99.44 

Navy 

FYOl 

FYOO 

FY99 

FY98 

FY97 

FY96 

FY95 

FY94 

FY93 

Avg 

StDev 

Yrl 

59.5 

59.3 

60.5 

58.0 

55.9 

55.9 

54.0 

55.0 

55.0 

57.01 

2.35 

Yr2 

31.4 

33.6 

32.5 

33.1 

31.5 

31.5 

32.4 

33.4 

33.4 

32.53 

0.89 

Yr3 

5.9 

4.5 

4.5 

5.4 

8.2 

8.2 

8.0 

7.8 

7.8 

6.70 

1.61 

Yr4 

1.9 

1.0 

1.0 

2.0 

2.0 

2.0 

2.0 

1.3 

1.3 

1.61 

0.45 

Yr5 

0.7 

0.3 

0.3 

0.6 

1.1 

1.1 

1.1 

1.1 

1.1 

0.82 

0.35 

Yr6 

0.0 

0.2 

0.2 

0.2 

0.4 

0.4 

0.4 

0.4 

0.4 

0.29 

0.15 

Yr7 

0.0 

0.0 

0.0 

0.0 

0.6 

0.0 

0.2 

0.0 

0.2 

0.11 

0.20 

99.4 

98.9 

99.0 

99.3 

99.7 

99.1 

98.1 

99.0 

99.2 

99.08 
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Table  4  shows  the  average  outlay  rates  and  the  standard  deviation  for  each  military 
branch  of  service.  Table  5  gives  an  illustration  of  our  calculations. 


Table  5.  RSIP  Final  Budget  Converted  to  Expenditures  (Current  $M) 


Average  Air  Force  Outlay  Rates  (1993-2001) 

Year 

Year  1 

Year  2 

Year  3 

Year  4 

Year  5 

Year  6 

■ 

Rate 

0.5156 

0.3664 

0.0714 

0.0269 

0.0081 

0.0031 

■ 

FY 

Budget 

1989 

1990 

1991 

1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

2000 

2001 

1989 

44.2 

22.8 

16.2 

3.2 

1.2 

0.4 

0.1 

1990 

63.7 

32.8 

23.3 

1.7 

0.5 

0.2 

1991 

71.8 

37.0 

26.3 

5.1 

1.9 

0.6 

0.2 

1992 

117.1 

60.4 

42.9 

8.4 

3.1 

0.9 

0.4 

1993 

15.4 

7.9 

5.6 

1.1 

0.4 

0.1 

0.0 

1994 

38.4 

19.8 

14.1 

2.7 

1.0 

0.3 

0.1 

1995 

42.7 

22.0 

15.6 

3.1 

1.1 

0.3 

0.1 

1996 

31.1 

16.0 

11.4 

2.2 

0.8 

0.3 

0.1 

22.8 

49.0 

63.5 

92.4 

58.1 

36.4 

41.1 

36.0 

16.0 

3.7 

1.3 

0.4 

0.1 

The  final  step  converts  the  program  current-dollar  expenditures  shown  as  the 
bottom  line  in  Table  5,  to  constant-dollar  expenditures  by  removing  the  effects  of 
inflation.  Table  6  shows  the  current  to  constant  dollar  conversion  with  (2).  The  current- 
dollar  expenditures  in  row  2  are  divided  by  the  raw  inflation  indices  in  row  3,  producing 
the  desired  constant-dollar  expenditures  in  row  4.  Figure  10  graphically  illustrates  the 
final  budget  profile,  expenditure  profile  in  current  dollars,  and  the  expenditure  profile  in 
constant  dollars  for  the  RSIP  program. 


Table  6.  RSIP  Current  $M  Expenditures  to  Constant  $M  Expenditures 


Fiscal  Year 

1989 

1990 

1991 

1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

2000 

2001 

Exp.  (Current  $M) 

22.8 

49.0 

63.5 

92.4 

58.1 

36.4 

41.1 

36.0 

16.0 

3.7 

1.3 

0.4 

0.1 

Inflation  Factor 

0.787 

0.818 

0.854 

0.877 

0.901 

0.919 

0.937 

0.955 

0.975 

0.982 

0.990 

1.000 

1.015 

Exp.  (Constant  $M) 

29.0 

59.9 

74.4 

105.3 

64.4 

39.6 

43.9 

37.7 

16.4 

3.8 

1.3 

0.4 

0.1 
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Figure  10.  RSIP  Budget  and  Expenditure  Profiles 

Table  7  provides  a  summary  of  our  conversion  calculations  with  (1)  and  (2). 


Table  7.  RSIP  Budget  to  Expenditures  Summary  Conversion 


Fiscal  Year 

1989 

1990 

1991 

1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

2000 

2001 

Budget  (Current  $M) 

44.2 

63.7 

71.8 

117.1 

15.4 

38.4 

42.7 

31.1 

Exp.  (Current  $M) 

22.8 

49.0 

63.5 

92.4 

58.1 

36.4 

41.1 

36.0 

16.0 

3.7 

1.3 

0.4 

0.1 

29.0 

59.9 

74.4 

105.3 

64.4 

39.6 

43.9 

37.7 

16.4 

3.8 

1.3 

0.4 

0.1 

In  summary,  we  convert  a  budget  profile  to  an  expenditure  profile  in  constant 
dollars.  We  now  turn  our  attention  to  the  estimation  of  the  Weibull  parameters.  We  use 
the  least  squares  error  method  by  fitting  the  Weibull  cumulative  distribution  of 
expenditures  to  the  RSIP  cumulative  constant  dollar  expenditures.  We  test  our 
assumption  that  R&D  program  expenditures  are  Weibull  distributed  using  GOT  statistics. 
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Estimate  Weibull  Parameters 


Nonlinear  estimation  is  the  method  we  used  to  estimate  the  Weibull  shape,  scale, 
and  location  parameters  in  (5).  We  allow  the  Weibull  parameters  to  vary  until  we 
minimize  the  sum-squared  difference  between  the  actual  cumulative  constant-dollar 
expenditures  and  the  Weibull  modeled  cumulative  constant-dollar  expenditures.  From  a 
budget  profile,  the  actual  cumulative  constant-dollar  expenditures  are  calculated  with  (1) 
and  (2).  The  Weibull  cumulative  constant-dollar  expenditures  are  calculated  with  (5). 

We  use  Microsoft  Excel’s  Solver  package  (2000)  as  our  nonlinear  estimation  tool. 
The  target  cell  in  Solver  is  set  to  minimize  the  sum  of  all  the  errors  squared.  The  Weibull 
shape,  scale,  and  location  parameters  are  set  as  the  ehanging  cells.  Solver  determines  the 
optimal  values  for  the  Weibull  shape,  seale,  and  loeation  parameters  that  minimize  the 
total  sum-squared  errors  with  (14). 

Table  8  shows  the  eonversion  proeess  from  the  reported  final  budget  to 
eumulative  eonstant-dollar  expenditures  for  the  Air  Force  RSIP  program.  The  final 
budget  profile,  eonversion  to  eurrent-dollar  expenditures,  conversion  to  constant-dollar 
expenditures,  and  resulting  eumulative  eonstant-dollar  expenditures  are  displayed  in  rows 
2,  3,  4,  and  5  respectively. 


Table  8.  RSIP  Cumulative  Constant-Dollar  Expenditure  Profile 


Fiscal  Year 

1989 

1990 

1991 

1992 

1993 

1994 

1995 

1996 

1997 

1998 

1999 

2000 

2001 

Budget  (Current  $M) 

44.2 

63.7 

71.8 

117.1 

15.4 

38.4 

42.7 

31.1 

Exp.  (Current  $M) 

22.8 

49.0 

63.5 

92.4 

58.1 

36.4 

41.1 

36.0 

16.0 

3.7 

1.3 

0.4 

0.1 

Exp.  (Constant  $M) 

29.0 

59.9 

74.4 

105.3 

64.4 

39.6 

43.9 

37.7 

16.4 

3.8 

1.3 

0.4 

0.1 

Cum.  Exp.  (Constant  $M) 

29.0 

163.3 

268.6 

333.0 

372.6 

416.5 

454.2 

470.6 

474.4 

475.7 

476.1 

476.2 
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We  now  estimate  the  Weibull  parameters  that  minimize  the  sum-squared  errors.  Table  9 
presents  the  minimization  outcome  for  the  RSIP  program.  The  total  sum-squared  error  is 
greater  than  510.1  for  any  other  combination  of  Weibull  parameters. 


Table  9.  RSIP  Minimized  E(errors)^ 


Fiscal 

Year 

Cumulative  CYOOSM 
Expenditures 

Error 

■ 

Actual 

Weibull 

1989 

29.0 

24.6 

4.4 

19.2 

1990 

88.9 

91.4 

-2.5 

6.5 

1991 

163.3 

174.0 

-10.7 

113.8 

1992 

268.6 

255.6 

13.0 

169.1 

1993 

333.0 

326.5 

6.6 

43.1 

1994 

372.6 

382.3 

-9.6 

92.9 

1995 

416.5 

422.8 

-6.3 

39.8 

1996 

454.2 

450.4 

3.9 

15.0 

1997 

470.6 

467.9 

2.7 

7.3 

1998 

474.4 

476.2 

-1.8 

3.2 

1999 

475.7 

476.2 

-0.5 

0.2 

2000 

476.1 

476.2 

-0.1 

0.0 

2001 

476.2 

476.2 

0.0 

2  1 

510.1 

1  —  | 

Table  10  presents  the  estimated  Weibull  parameters  that  produce  our  minimization 
results.  The  cost  factor  is  equal  to  the  final  cost  of  the  program  divided  by  0.97.  The 
initial  search  values  for  the  shape  and  scale  parameters  are  set  greater  than  0.1.  The 
location  parameter  is  set  to  zero  to  represent  presumed  immediate  program  start. 

Table  10.  RSIP  Least  Squares  Estimated  Weibull  Parameters 


Cost  Factor 

Scale  5 

WIWWilB 

490.93 

4.540 

1.694 

0.213 
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Figure  1 1  demonstrates  the  resulting  Weibull  modeled  constant-dollar 


expenditures  fit  to  the  cumulative  constant-dollar  expenditures. 


Figure  12  shows  the  resulting  Weibull  modeled  constant-dollar  expenditure  fit  to  the 
cumulative  constant-dollar  expenditures  converted  to  annual  expenditures. 


Figure  12.  RSIP  Cumulative  Least  Squares  Weibull  Fit  Converted  to  Annual 
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In  summary,  minimizing  the  total  sum-squared  difference  between  the  actual 
cumulative  constant-dollar  expenditures  and  the  Weibull  modeled  cumulative  constant- 
dollar  expenditures  provides  the  Weibull  estimated  parameters.  The  estimated  Weibull 
shape  and  scale  parameters  provide  the  responses  or  dependent  variables  for  our 
regression  analysis.  The  categorical  and  the  final  cost  and  schedule  data  provide  the 
predictors  or  independent  variables  for  our  regression  model. 

Perform  Goodness-of-Fit  (GOF)  Tests 

Porter  (2001)  and  Unger  (2001)  find  that  the  Weibull  model  better  fits  R&D 
program  expenditures.  The  measurements  we  use  to  evaluate  if  expenditures  follow  the 
theoretical  Weibull  are  three  goodness-of-fit  (GOF)  statistics,  the  Komolgorov-Smimov 
(K-S),  Cramer-von  Mises  (CvM),  and  Anderson-Darling  (A-D).  Unger  (2001)  modifies 
each  of  these  statistics  to  perform  GOF  tests  on  discrete  distributions.  Unger  statistically 
compares  the  deviation  between  F„(x),  which  is  the  empirical  distribution  function  (EDF) 
based  on  n  data  points,  and  Fo{x),  which  is  the  hypothesized  CDF.  Using  Unger’s 
application,  the  EDF  is  the  cumulative  constant-year  expenditures  reported  at  the  end  of 
each  fiscal  year,  from  (13),  divided  by  the  cost  factor,  F^{i)  =  C,.  / d.  The  hypothesized 

model  is  (1)  with  the  least  squares  estimates  for  the  Weibull  from  (14).  Unger  defines  the 
K-S,  CvM,  and  A-D  GOF  statistics  as 

K  =  max|F„ (/)  - F^ (/)|  for  i  =  1,2,..., N  +  J  -\,  (15) 

AD=r%{FM"HFSi  +  m-^^im))- 

/=! 

(F„(/-l)-l)^(ln(l-F„(/))-ln(l-F„(/-l)))]-«,  (16) 
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and 


,  /=«+Ir 


^  /=! 


(17) 


respectively.  Unger  provides  a  description  and  the  derivations  for  the  K-S,  CvM,  and  A- 
D  GOF  statistics. 


Regression  Analysis 

We  use  multiple  regression  to  determine  if  there  is  a  mathematical  relationship 
between  the  necessary  Weibull  parameters  and  possible  predictors  that  a  new  start  R&D 
program  possesses.  To  build  the  Weibull  shape  and  scale  regression  models,  we  select 
two  categorical  (nominal)  and  two  numerical  (continuous)  predictor  variables.  The  two 
categorical  predictors  are  program  lead  branch  of  service  (Air  Force,  Army,  and  Navy) 
and  program  system  type  (Aircraft,  Electronic,  Missile,  Munitions,  Ship,  Space,  and 
Vehicle).  The  lead  service  for  joint  programs  is  determined  by  the  service  that 
contributed  the  highest  dollar  value  to  the  R&D  effort.  The  two  numerical  predictors  are 
the  program  total  2000  constant  dollars  (CYOOS)  and  the  program  duration  in  years  to 
Milestone  III.  Using  the  predictive  values  from  our  shape  and  scale  regression  models, 
we  apply  the  Weibull  model  to  forecast  R&D  program  budget  profiles.  This  leads  to  the 
final  section  of  our  methodology  of  relating  the  Weibull  model  to  budget  profiles. 

Convert  Expenditure  Profiles  to  Budget  Profiles 

The  shape  and  scale  are  the  necessary  parameters  to  utilize  the  Weibull  model  to 
forecast  R&D  program  budget  profiles.  The  Weibull  shape  and  scale  predictive  models 
we  obtain  from  our  regression  analysis  provide  the  necessary  predicted  parameters  to 
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employ  the  Weibull  model.  Lee,  Hogue,  and  Gallagher  (1997)  forecast  a  budget  profile 
from  a  point  estimate  using  Rayleigh-based  expenditures.  We  employ  their  method  using 
Weibull-based  expenditures. 

Chapter  Summary 

This  chapter  explains  the  proposed  methodology  to  predict  the  requisite  shape  and 
scale  parameters  to  forecast  Weibull-based  initial  R&D  program  budget  profiles.  We 
search  for  relationships  to  predict  the  Weibull  shape  and  scale  parameters  using  multiple 
regression  analysis.  We  convert  128  program’s  current-dollar  budget  profiles  to  fiscal 
year  2000  constant  dollar  cumulative  expenditure  profiles.  To  build  the  response  portion 
of  our  model  data,  we  estimate  the  shape  and  scale  parameters  by  minimizing  the  sum 
errors  squared  between  the  actual  program  cumulative  expenditures  and  the  Weibull 
modeled  cumulative  expenditures.  We  randomly  selected  102  (80%)  of  the  programs  to 
perform  our  regression  analysis,  and  the  remaining  26  (20%)  are  saved  for  model 
validation.  Finally,  we  forecast  Weibull-based  R&D  program  budget  profiles  by  using 
the  predicted  shape  and  scale  values  from  our  regression  models.  In  Chapter  Four,  we 
provide  the  results  of  our  methodology  application  to  completed  R&D  programs. 
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IV.  Research  Results 


Chapter  Overview 

Chapter  Three  outlined  the  methodology  to  predict  shape  and  scale  parameters 
requisite  to  forecast  Weibull-based  R&D  budget  profiles.  This  chapter  presents  our 
research  results  when  applying  the  128  completed  R&D  defense  programs  to  our 
methodology.  We  present  the  goodness-of-fit  (GOF)  results  in  support  of  our  assumption 
that  the  Weibull  model  fits  R&D  program  expenditures.  We  present  our  Weibull  shape 
and  scale  predictive  models,  and  demonstrate  that  both  regression  models  are  statistically 
significant.  Using  our  predictive  shape  and  scale  models  to  utilize  the  Weibull  function, 
we  demonstrate  the  Weibull  model’s  ability  to  forecast  budget  profiles.  Finally,  we 
demonstrate  the  improved  foreeasting  ability  of  the  Weibull  model  when  compared  to 
that  of  the  Rayleigh  model. 

Goodness  of  Fit  Results 

The  three  GOF  statistics  we  use  to  determine  if  the  128  R&D  program 
expenditures  are  Weibull  distributed  are  the  Komolgorov-Smimov  (K-S),  Cramer-von 
Mises  (CvM),  and  Anderson-Darling  (A-D).  The  test  results  are  separated  by  the 
program  budget  duration  in  years.  Programs  that  are  accepted  represent  expenditures  that 
are  Weibull  distributed.  Tables  1 1-13  show  the  results  for  the  K-S  (89%  accept),  CvM 
(71%  accept),  and  A-D  (72%  accept)  GOF  tests  performed  for  all  128  programs 
respectively. 
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Table  11.  Komolgorov-Smirnov  Goodness-of-Fit  Results 

Program  Duration 

Programs 

Accept 

Reject 

%  Accept 

%  Reject 

Duration  <  3 

5 

3 

2 

3  <  Duration  <  4 

17 

11 

6 

65% 

4  <  Duration  <  5 

15 

14 

1 

93% 

7% 

5  <  Duration  <  6 

14 

14 

0 

100% 

0% 

6  <  Duration  <  7 

14 

12 

2 

86% 

14% 

7  <  Duration  <  22 

63 

60 

3 

95% 

5% 

Total 

128 

114 

14 

89% 

11% 

Table  12. 

Cramer-von  Mises  Goodness-of-Fit  Results 

Program  Duration 

Programs 

Accept 

Reject 

%  Accept 

%  Reject 

Duration  <  3 

5 

0 

5 

0% 

100% 

3  <  Duration  <  4 

17 

1 

16 

6% 

94% 

4  <  Duration  <  5 

15 

7 

8 

47% 

53% 

5  <  Duration  <  6 

14 

10 

4 

71% 

29% 

6  <  Duration  <  7 

14 

13 

1 

93% 

7% 

7  <  Duration  <  22 

63 

60 

3 

95% 

5% 

Total 

128 

91 

37 

71% 

29% 

Table  13. 

Anderson-Darling  Goodness-of-Fit  Results 

Program  Duration 

Programs 

Accept 

Reject 

%  Accept 

%  Reject 

Duration  <  3 

5 

0 

5 

0% 

100% 

3  <  Duration  <  4 

17 

7 

10 

41% 

59% 

4  <  Duration  <  5 

15 

11 

4 

73% 

27% 

5  <  Duration  <  6 

14 

8 

6 

57% 

43% 

6  <  Duration  <  7 

14 

10 

4 

71% 

29% 

7  <  Duration  <  22 

63 

56 

7 

89% 

11% 

Total 

128 

92 

36 

72% 

28% 

Table  14  shows  77  percent  of  the  128  program  expenditure  profiles  are  Weibull 
distributed.  The  GOF  test  results  to  determine  if  R&D  program  expenditures  are  Weibull 
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distributed  appears  lower  than  expected;  however,  annual  R&D  program  expenditures 
represent  a  discrete  distribution.  Many  continuous  GOF  tests  fail  discrete  distributions 
because  of  the  infinite  number  of  points  between  each  discrete  value.  Unless  the  discrete 
distribution  is  highly  populated  as  to  approach  a  continuous  distribution  the  GOF  test  will 
fail.  Unger  (2001)  adjusts  the  K-S,  CvM,  and  A-D  formulas  to  perform  GOF  statistics  on 
program  expenditures  that  are  discretely  distributed.  The  GOF  tests  do  not  penalize  the 
expenditure  profile  any  more  or  less  than  the  previous  expenditure  year.  This  in  essence 
converts  the  expenditure  profile  into  a  rigid  continuous  distribution  increasing  or 
decreasing  with  each  expenditure  year.  The  more  rigid  the  expenditure  profile  the  less 
likely  the  program  expenditures  are  Weibull  distributed.  Smoother  expenditure  profiles 
are  more  likely  for  programs  with  many  expenditure  years,  thus  we  expect  higher  failure 
rates  for  smaller  program  durations. 


Table  14.  Overall  Goodness-of-Fit  Test  Results 


Test  Type 

Accept 

Reject 

%  Accept 

%  Reject 

Komolgorov-Smimov  (K-S) 

114 

14 

89% 

11% 

Cramer-von  Mises  (CvM) 

91 

37 

71% 

29% 

Anderson-Darling  (A-D) 

92 

36 

72% 

28% 

Total 

297 

87 

77% 

23% 

The  GOF  results  show  high  failure  rates  for  the  5 1  programs  with  duration  of  six 
years  or  less.  Table  15  indicates  that  only  56  percent  of  these  51  program  expenditures 
are  Weibull  distributed.  However,  if  monthly  expenditure  data  were  available  for  the  51 
programs  with  duration  six  years  or  less,  we  contend  that  90  percent  of  the  program 
monthly  expenditures  are  Weibull  distributed.  Table  16  indicates  that  91  percent  of  the 
77  programs  with  duration  greater  than  six  years  are  Weibull  distributed. 
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Table  15.  Goodness-of-Fit  Results  (Budget  Profiles  <=  6  Years) 


Test  Type  (51  Programs) 

Accept 

Reject 

%  Accept 

%  Reject 

Komolgorov- Smirnov  (K-S) 

42 

9 

82% 

18% 

Cramer-von  Mises  (CvM) 

18 

33 

35% 

65% 

Anderson-Darling  (A-D) 

26 

25 

51% 

49% 

Total 

86 

67 

56% 

44% 

Table  16.  Goodness-of-Fit  Results  (Budget  Profiles  >  6  Years) 


Test  Type  (77  Programs) 

Accept 

Reject 

%  Accept 

%  Reject 

Komolgorov- Smirnov  (K-S) 

72 

5 

94% 

6% 

Cramer-von  Mises  (CvM) 

73 

4 

95% 

5% 

Anderson-Darling  (A-D) 

66 

11 

86% 

14% 

Total 

211 

20 

91% 

9% 

The  GOF  statistics  supports  the  assumption  that  R&D  program  expenditures  are 
Weibull  distributed.  The  next  section  reports  the  results  of  our  regression  analysis  to  test 
for  relationships  that  predict  the  Weibull  scale  and  shape  parameters. 


Weibull  Scale  and  Shape  Regression  Models 

Research  and  Development  (R&D)  defense  program  expenditures  are  Weibull 
distributed.  To  apply  Lee,  Hogue,  and  Gallagher’s  (1997)  method  to  forecast  a  budget 
profde  from  a  point  estimate  using  the  Weibull  model,  we  must  predict  the  scale  and 
shape  parameters  with  information  known  to  new  start  R&D  programs.  Appendix  A 
shows  the  128  completed  R&D  programs  we  use  to  determine  a  relationship  that  predicts 
the  necessary  Weibull  scale  and  shape  parameters.  This  section  of  the  results  will  present 
the  scale  model,  the  shape  model,  and  the  validation  results  in  predicting  the  scale  and 
shape  least  squares  parameter  estimates. 


41 


Regression  Analysis  Setup.  We  randomly  select  102  (80%)  of  the  total  128 
programs  to  test  for  mathematical  relationships  for  each  of  the  Weibull  scale  and  shape 
parameters.  The  response  or  dependent  variables  for  each  regression  model  are  the 
Weibull  shape  and  scale  least  squares  parameter  estimates  from  Solver.  We  use  four 
possible  predictors  or  independent  variables.  Branch  of  service  and  program  system  type 
are  the  first  two  categorical  predictors  converted  from  nominal  data  to  continuous  data 
via  indicator  variables.  We  set  Air  Force  as  the  baseline  and  convert  the  branch  of 
service  column  data  into  two  separate  predictor  columns  as  indicator  variables  for  Army 
and  Navy.  For  the  Army  data  column,  we  assign  a  one  for  Army  programs  and  a  zero  for 
Air  Force  and  Navy  programs.  For  the  Navy  data  column,  we  assign  a  one  for  Navy 
programs  and  a  zero  for  Air  Force  and  Army  programs.  We  set  Vehicle  as  the  baseline 
for  the  second  categorical  predictor,  program  system  type.  We  establish  a  data  column 
for  each  of  the  remaining  program  system  type  categories  (Aircraft,  Electronic,  Missile, 
Munitions,  Ship  and  Space)  in  the  same  fashion  as  Army  and  Navy  data  columns. 
Program  total  cost  in  fiscal  year  2000  constant  dollars  divided  by  0.97  (cost  factor,  d)  and 
program  duration  in  total  budget  years  minus  the  location  estimate  are  the  final  two 
continuous  predictor  variables.  We  use  multiple  regression  to  find  a  relationship  for  the 
Weibull  scale  and  shape  parameters. 

The  Weibull  Scale  Model.  For  the  scale  model  results,  we  display  graphically  the 
least  squares  estimated  scale  via  Solver  by  the  model  predicted  scale.  Statistically,  we 
present  the  summary  of  fit,  analysis  of  variance  (ANOVA),  and  parameter  model 
estimates.  Finally,  we  discuss  the  scale  model  as  a  statistically  significant  predictor  for 
the  Weibull  scale  parameter. 
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Figure  13  displays  the  least  squares  estimated  scale  by  the  model’s  predicted 
scale.  The  figure  clearly  shows  that  the  least  squares  estimated  scale  values  are  tightly 
arranged  along  the  model’s  predicted  scale  regression  line,  indicating  that  our  regression 
model  predicts  scale  well. 


Figure  13.  Least  Squares  Estimated  Scale  by  Predicted  Scale  Plot 

We  select  the  adjusted  R  over  R  as  the  statistical  measure  of  the  model’s  ability 
to  predict  the  response  (least  squares  estimated  scale).  The  adjusted  R  compares  across 
models  with  different  numbers  of  parameters  by  using  the  degrees  of  freedom  in  its 
computation  (Sail,  Lehman,  and  Creighton,  2001).  In  our  analysis  of  several  models  we 
find  that  no  additional  predictor  variables  or  interactions,  other  than  program  duration, 
significantly  improve  the  adjusted  R  of  0.921. 

The  ANOVA  displays  the  overall  F  test,  indicating  a  significant  model.  When  the 
significance  level  or  /i-value  is  less  than  0.05  the  overall  model  is  significant,  indicating 
that  at  least  one  predictor  is  significant.  Because  program  duration  is  our  only  predictor 
of  scale,  the  t-test,  which  measures  the  significance  of  each  predictor  in  the  model,  will 
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mirror  the  overall  F  test.  Table  17  shows  the  summary  of  fit,  the  AN OVA,  and  parameter 
estimates.  We  define  our  final  scale  regression  model,  <^hat,  as 

d  =  0.126{Dumtion) .  (18) 

Table  17,  Scale  Model  Table  of  Statistics 

Scale  Model  Summary  of  Fit _ 

RSquare 
RSquare  Adj 

Root  Mean  Square  Error 
Mean  of  Response 
Observations  (or  Sum  Wgts) 

Scale  Model  Analysis  of  Variance  (AN OVA) 


Source 

Model 

Error 

C.  Total 

DF 

1 

100 

101 

Sum  of  Squares 
799.75149 

67.96724 

867.71873 

Mean  Square 
799.751 

0.680 

F  Ratio 

1176.672 

Prob  >  F 

<.0001 

Scale  Model  Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>  t 

Intercept 

0.0683049 

0.187391 

0.36 

0.7163 

Duration 

0.7256199 

0.021153 

34.30 

<.0001 

To  test  the  statistical  soundness  of  our  scale  regression  model,  we  look  for 
possible  influential  data  points  that  bias  what  explanatory  variables  are  selected,  and  test 
the  model  assumptions  of  normality,  constant  variance,  and  independence. 

Scale  Model  Influential  Data  Test.  We  test  for  possible  influential  data  points  by 
plotting  Cook’s  D  influence  statistic.  Cook’s  D  influential  statistic  determines  if  an 
observation  has  a  large  effect  on  parameter  estimates.  Values  greater  than  0.5  are 
considered  significant  influential  observations  (Neter,  Kutner,  Nachtsheim,  and 


0.921671 

0.920888 

0.824422 

5.854373 

102 
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Wasserman,  1996:380).  Figure  14  displays  the  overlay  plot  of  values  for  Cook’s  D 
influential  statistic,  indicating  that  no  observations  are  significantly  influential. 


Scale  Model  Normality  Test.  We  test  for  normality  by  performing  a  GOF  test  to 
determine  if  the  residuals  from  our  regression  model  are  normally  distributed.  Figure  15 
shows  the  normal  distribution  fit  to  the  residual  distribution  and  displays  the  normality 
test  results.  If  the  /i-value  is  greater  than  0.05  then  the  residuals  are  normally  distributed. 
We  display  a /i-value  of  0.901  indicating  that  the  model  residuals  are  normally 
distributed. 
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Figure  15.  Scale  Model  Normality  Test 


Scale  Model  Constant  Variance  Test.  We  test  for  constant  variance  by  plotting 
the  residuals  by  the  predicted  scale  values.  We  visually  conclude  constant  variance  if  the 
plotted  values  are  uniformly  distributed.  Figure  16  displays  a  reasonably  uniform 
distribution  of  values  for  the  residual  by  predicted  plot  indicating  that  our  model  has 
constant  variance. 


Figure  16.  Scale  Model  Test  for  Constant  Variance 
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Scale  Model  Independence  of  Data  Test.  The  test  for  independence  is  not 
performed.  We  assume  independence  in  our  data  because  we  do  not  duplicate  selected 
programs.  Thus,  no  obvious  serial  pattern  exists  to  test  for  independence. 

The  Weibull  Shape  Model.  For  the  shape  model  results,  we  display  graphically 
the  least  squares  estimated  shape  by  model  predicted  shape.  Statistically,  we  present  the 
summary  of  fit,  analysis  of  variance,  and  parameter  model  estimates.  Finally,  we  discuss 
the  shape  model  as  a  statistically  significant  predictor  for  the  Weibull  shape  parameter. 

Figure  17  displays  the  least  squares  estimated  shape  by  the  model’s  predicted 
shape.  This  figure  clearly  shows  that  the  least  squares  estimated  shape  values  are  not  as 
correlated  with  our  model’s  predicted  shape  in  comparison  to  the  scale  model.  This 
indicates  that  our  shape  regression  model  does  not  appear  to  predict  shape  as  well  as  our 
scale  model  predicts  scale. 


Figure  17.  Least  Squares  Estimated  Shape  by  Predicted  Shape  Plot 
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Table  18,  Shape  Model  Table  of  Statistics 


Shape  Model  Summary  of  Fit 

RSquare 

0.310116 

RSquare  Adj 

0.274185 

Root  Mean  Square  Error 

0.763702 

Mean  of  Response 

2.724529 

Observations  (or  Sum  Wgts) 

102 

Shape  Model  Analysis  of  Variance  (ANOVA) 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

5 

25.169127 

5.03383 

8.6308 

Error 

96 

55.991124 

0.58324 

Prob  >  F 

C.  Total 

101 

81.160251 

<.0001 

Shape  Model  Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>t 

Intercept 

1.299561 

0.32514 

4.00 

nn»»n 

ln(l /Duration) 

-0.973254 

0.16037 

-6.07 

Army 

-0.423434 

0.20643 

-2.05 

Navy 

-0.485661 

0.18882 

-2.57 

0.0116 

Electronic 

-0.545079 

0.18152 

-3.00 

Space 

-1.100189 

0.56290 

-1.95 

0.0536 

Table  18  displays  the  summary  of  fit,  ANOVA,  and  parameter  model  estimates. 
The  adjusted  R  m  the  summary  of  fit  table,  0.27,  appears  to  indicate  that  our  model  is 
not  a  good  fit  for  predicting  shape.  However,  we  show  later  that  our  shape  model 
predicts  the  least  squares  estimated  shape  well  enough  to  employ  the  Weibull  model  with 
significant  results.  The  overall  F  test  in  the  ANOVA  table  shows  a  /7-value  less  than 
0.05,  indicating  that  our  overall  shape  model  is  significant.  The  t-test  in  the  parameter 
estimates  table  shows  /7-values  less  than  0.05  for  all  predictors  except  Space.  When  the 
/7-value  is  greater  than  0.05,  the  predictor  is  statistically  insignificant.  Space  is 
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statistically  insignificant  but  not  enough  to  give  up  to  the  additional  predictability  it  adds 
to  the  overall  shape  model.  Our  shape  model  does  possess  one  transformation,  which 
implies  that  there  is  a  better  mathematical  relationship  between  ln(l /Duration)  and  shape 
than  duration  simply  itself  We  define  the  final  shape  regression  model,  fi-hdX,  as 

0  =  (19) 

where  P,  values  are  the  parameter  estimates  and  Xj  are  the  shape  model  parameter  terms 
listed  in  order  in  Table  18. 

To  test  the  statistical  soundness  of  our  shape  regression  model,  we  look  for 
possible  influential  data  points  that  bias  what  explanatory  variables  are  selected,  and  test 
the  model  assumptions  of  normality,  constant  variance,  and  independence. 


Shape  Model  Influential  Data  Test.  Figure  18  displays  the  overlay  plot  of  values 
for  Cook’s  D  influential  statistie,  indicating  that  no  observations  are  significantly 
influential. 


Figure  18.  Shape  Model  Test  for  Influential  Data  Points 
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Shape  Model  Normality  Test.  In  testing  the  normality  assumption,  Figure  19 
reveals  that  the  residuals  are  barely  normally  distributed,  with  a  /7-value  of  0.0802. 


Figure  19.  Shape  Model  Normality  Test 


However,  this  is  potentially  misleading  because  of  one  skewed  data  point  displayed  in 
Figure  19  in  the  box  and  whiskers  plot  as  a  dot. 
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Shape  Model  Constant  Variance  Test.  Figure  20  shows  a  reasonably  uniform 
distribution  of  values  for  the  residual  by  predicted  plot  indicating  that  our  shape  model 
has  constant  variance. 

Shape  Model  Independence  of  Data  Test.  The  test  for  independence  is  not 
performed.  We  assume  independence  in  our  data  because  we  do  not  duplicate  selected 
programs.  Thus,  no  obvious  serial  pattern  exists  to  test  for  independence. 

Scale  and  Shape  Model  Validation 

We  show  that  both  the  scale  and  shape  models  are  statistically  significant.  The 
overall  F  test  and  the  individual  parameter  t-tests  indicate  that  both  overall  models  and 
individual  predietors  within  eaeh  model  report p-values  less  than  0.05.  To  validate  the 
robustness  of  our  shape  and  seale  regression  models,  we  show  that  1 00  and  96  percent  of 
the  remaining  26  (20%  of  full  data  set)  eompleted  R&D  program  least  squares  estimated 
shape  and  seale  validation  data  values  fall  within  a  95  percent  prediction  interval.  Our 
validation  results  demonstrate  that  we  did  not  over-fit  the  data  to  build  the  shape  and 
scale  regression  models.  Table  19  and  Table  20  display  the  95  percent  prediction  interval 
upper  and  lower  bounds,  the  least  squares  estimated  scale  and  shape  values,  and  the 
accept/reject  results  for  the  scale  and  shape  regression  models  respectively. 


51 


Table  19.  Scale  Model  Validation  Using  a  95%  Prediction  Interval 


_#_ 

Program 

LB 

Scale 

Actual 

Scale 

UB 

Scale 

Reject/ 

Accept 

1 

ADDS 

4.230 

5.683 

7.517 

Accept 

2 

ALCM 

2.776 

4.764 

6.068 

Accept 

3 

ASM  LOSAT 

4.230 

6.671 

7.517 

Accept 

4 

ATACMS 

6.402 

8.432 

9.699 

Accept 

5 

ATACMS-APAM 

1.319 

2.521 

4.623 

Accept 

6 

AVENGER 

0.588 

2.546 

3.902 

Accept 

7 

B-2 

6.777 

7.323 

10.078 

Accept 

8 

BATTEESHIP 

2.776 

4.017 

6.068 

Accept 

9 

C/MH-53E 

2.753 

3.930 

6.044 

Accept 

10 

E-6A 

2.228 

4.568 

5.524 

Accept 

11 

F-14 

1.104 

2.572 

4.411 

Accept 

12 

GECM 

2.776 

4.428 

6.068 

Accept 

13 

HH-60D 

2.048 

3.619 

5.345 

Accept 

14 

JAVELIN 

7.123 

6.583 

10.428 

Reject 

15 

LANTIRN 

3.504 

5.631 

6.792 

Accept 

16 

LASER  HELLFIRE 

6.402 

8.553 

9.699 

Accept 

17 

LBG 

2.048 

3.532 

5.345 

Accept 

18 

MMIII  GRP 

3.504 

4.571 

6.792 

Accept 

19 

NAVSTAR  GPS 

7.844 

9.900 

11.158 

Accept 

20 

PATRIOT(Missile  Seg) 

12.148 

14.683 

15.563 

Accept 

21 

PLSS 

7.844 

11.017 

11.158 

Accept 

22 

ROTHR 

3.190 

4.337 

6.480 

Accept 

23 

RPV 

8.564 

11.186 

11.890 

Accept 

24 

SCAMP 

1.319 

3.499 

4.623 

Accept 

25 

SINCGARS 

3.813 

5.573 

7.100 

Accept 

26 

SSN21 

4.230 

7.002 

7.517 

Accept 
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Table  20.  Shape  Model  Validation  Using  a  95%  Prediction  Interval 


_#_ 

Program 

H 

Reject/ 

Accept 

1 

ADDS 

0.787 

2.098 

3.923 

Accept 

2 

ALCM 

1.495 

3.322 

4.592 

Accept 

3 

ASM  LOSAT 

1.353 

3.718 

4.447 

Accept 

4 

ATACMS 

1.657 

3.512 

4.763 

Accept 

5 

ATACMS-APAM 

0.670 

1.931 

3.780 

Accept 

6 

AVENGER 

0.378 

1.957 

3.513 

Accept 

7 

B-2 

2.115 

2.191 

5.241 

Accept 

8 

BATTEESHIP 

1.022 

1.930 

4.094 

Accept 

9 

C/MH-53E 

1.016 

1.915 

4.089 

Accept 

10 

E-6A 

0.312 

3.289 

3.452 

Accept 

11 

F-14 

0.540 

1.871 

3.638 

Accept 

12 

GECM 

1.495 

2.678 

4.592 

Accept 

13 

HH-60D 

1.317 

2.521 

4.415 

Accept 

14 

JAVELIN 

1.738 

2.273 

4.851 

Accept 

15 

LANTIRN 

1.096 

2.764 

4.201 

Accept 

16 

LASER  HELLFIRE 

1.657 

3.769 

4.763 

Accept 

17 

LBG 

1.317 

2.129 

4.415 

Accept 

18 

MMIII  GRP 

1.644 

2.207 

4.743 

Accept 

19 

NAVSTAR  GPS 

1.690 

2.429 

4.812 

Accept 

20 

PATRIOT  (Missile  Seg) 

2.112 

3.845 

5.247 

Accept 

21 

PLSS 

1.690 

4.094 

4.812 

Accept 

22 

ROTHR 

0.537 

2.371 

3.664 

Accept 

23 

RPV 

1.882 

4.043 

5.007 

Accept 

24 

SCAMP 

-0.785 

2.775 

3.035 

Accept 

25 

SINCGARS 

0.714 

2.967 

3.851 

Accept 

26 

SSN21 

1.302 

4.310 

4.374 

Accept 

The  prediction  interval  calculation  does  not  know  to  bound  shape  parameter  values  to 
strictly  greater  than  zero.  Thus,  we  note  that  data  point  24,  in  Table  20,  reports  an 
unrealistic  negative  lower  bound  shape  value. 
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Now  that  predictive  models  for  the  Weibull  shape  and  scale  parameters  are 
established  we  can  employ  the  Weibull  model.  In  the  next  section  we  evaluate  the 
capability  of  the  Weibull  model  to  forecast  completed  R&D  program  budget  profiles. 

Weibull  Model  Prediction  Capability  to  Actual  Budget  Profiles 

To  this  point  we  show  that  R&D  expenditures  are  Weibull  distributed  with  our 
GOF  results.  Our  statistical  analysis  shows  that  our  shape  and  scale  regression  models 
are  significant  and  predict  the  least  squares  estimated  shape  and  scale  values  well.  In  this 
section  we  discuss  how  we  employ  Lee,  Hogue,  and  Gallagher’s  (1997)  method  to 
forecast  R&D  program  budget  profiles  given  a  point  estimate  when  applying  the  Weibull 
model. 

We  convert  completed  R&D  program  budget  profiles  to  Fiscal  Year  2000 
constant  dollar  expenditure  profiles  using  (1)  and  (2).  We  sum  each  program’s  constant 
dollar  expenditure  profile  to  establish  a  program  point  estimate,  D  in  CY00$.  We  convert 
the  total  program  cost  estimate  to  a  Weibull  modeled  constant  dollar  expenditure  profile 
with  (5)  using  the  regression  models  to  predict  the  scale  and  shape  with  (18)  and  (19). 

We  convert  the  constant  dollar  expenditure  profile  to  current  expenditures  by  multiplying 
the  appropriate  raw  inflation  indices  found  in  Appendix  B.  Finally,  we  utilize  Lee, 
Hogue,  and  Gallagher’s  (1997)  method  of  constrained  nonlinear  estimation  to  develop  a 

budget  profile.  This  method  allows  the  budget  estimates,  B.  for  each  budget  year  i,  to 
change  simultaneously  until  an  optimal  solution  is  reached.  We  apply  this  approach  by 
substituting  the  5;  in  (1)  and  (2)  and  allow  Microsoft  Excel  Solver  function  (2000)  to 
select  the  yearly  budget  estimates  that  minimize  the  sum  of  squared  errors  with  (14)  to 
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the  end  of  budget  completion  between  the  Weibull  modeled  expenditures  and  the  actual 
expenditures  that  result  from  the  forecasted  budget  profile. 

As  a  note,  we  minimize  the  sum  of  squared  errors  to  the  end  of  budget  completion 
instead  of  to  the  end  of  expenditure  completion  to  produce  a  more  logical  program  budget 
profile.  This  slight  change  to  Lee,  Hogue,  and  Gallagher’s  (1997)  methodology  prevents 
Solver  from  producing  unrealistic  spikes  at  the  end  of  a  Weibull  model  forecasted  budget 
profile.  The  methodology  change  does  not  improve  our  results  to  forecast  actual  budget 
profiles. 

To  determine  the  capability  of  the  Weibull  model  to  forecast  budget  profiles,  we 
compare  the  forecasted  Weibull-based  budget  profiles  to  the  128  completed  R&D 
program  budget  profiles.  We  determine  the  eorrelation  between  each  completed  budget 
profile  and  the  Weibull  foreeasted  budget  profile.  To  determine  the  overall  Weibull 
model  effeetiveness  we  average  the  128  eorrelation  ealculations  between  the  completed 
(Aetual)  and  Weibull  (Foreeasted)  budget  profiles.  Table  21  summaries  our  results. 


Table  21.  Actual  by  Weibull-Based  Budget  Profile  Correlation  Breakdown 


Correlation  (c) 
Range 

Correlation 

Distribution 

%  Correlation 

Distribution 

c<0.5 

37 

29% 

0.5  <  c  <  0.6 

12 

9% 

0.6<c<0.7 

17 

13% 

0.7<c<0.8 

23 

18% 

0.8<c<0.9 

19 

15% 

0.9  <c<  1.0 

20 

16% 

Total 

128 

100% 

Average  Correlation 

0.6068 

Minimum  Correlation 

-0.9984 

Maximum  Correlation 

0.9986 
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We  report  an  average  correlation  of  0.607  for  the  128  programs.  This  shows  that  on 
average  61  percent  of  the  forecasted  Weibull-based  budget  profiles  fits  the  128  completed 
program  budget  profiles. 

We  find  that  low  correlations  between  Weibull  projected  budgets  and  actual 
budgets  are  a  result  of  small  program  durations.  Table  22  displays  the  percent 
distribution  by  budget  year  duration  for  programs  with  correlations  less  than  0.5.  As 
stated  earlier,  our  GOF  results  show  that  only  56  percent  of  expenditures  are  Weibull 
distributed  for  program  durations  less  than  seven  years  (small  discrete  distributions).  The 
results  show  that  roughly  59  percent  of  forecasted  Weibull-based  budgets  are  less  than  50 
percent  correlated  to  completed  programs  with  less  than  five  budget  years.  Thus, 
programs  with  durations  less  than  five  years  are  difficult  to  forecast  budget  profiles. 


Table  22.  Percent  Correlation  Distribution  by  Program  Duration 


Program 

Duration 

Program 

Distribution 

#  of  Programs 
w/Correlations 

Less  Than  0.5 

%  of  Programs 
w/Correlations 

Less  Than  0.5 

3 

5 

3 

60% 

4 

12 

7 

58% 

5 

19 

6 

32% 

6 

15 

6 

40% 

7 

12 

3 

25% 

8 

16 

4 

25% 

9 

9 

4 

44% 

>  10 

40 

4 

10% 

Summary 

Duration  <  7 

51 

22 

43% 

Duration  >  7 

77 

15 

19% 

Table  23  reports  that  4  percent  (5  of  the  128)  of  the  Weibull  modeled  budgets  are 
negatively  correlated  to  the  actual  completed  budgets.  The  negative  correlations  are 
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primarily  due  to  small  program  durations.  As  an  example,  Figure  21  displays  the  Army 
Avenger  combat  vehicle  final  program  budget,  and  the  forecasted  Weibull-based  budgets 
using  the  least  squares  estimates  for  the  shape  and  scale  parameters  via  Solver,  and  the 
predicted  shape  and  scale  parameters  via  our  regression  models.  We  show  that  a  negative 
correlation  exists  between  the  actual  budget  and  the  Weibull-based  budget  using  the  least 
squares  estimates  for  the  shape  and  scale  parameters,  indicating  that  the  Avenger 
expenditure  profile  is  not  Weibull  distributed.  We  note,  that  four  of  the  total  five 
programs  that  report  a  negative  correlation  have  less  than  five  budget  years. 
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'o 

Q 


o 

s 

o 


Fiscal  Year 


I  I  Actual  Budget  Profile 


—a— Weibull  w/  Least  Squares 
Estimated  Parameters 
Correlation  =  -0.797 

— Weibull  w/  Predicted 
Parameters 
Correlation  =  -0.998 


Figure  21.  Rayleigh-Based  Budget  Forecast  with  Negative  Correlation 


Weibull  and  Rayleigh  Model  Prediction  Comparison 

Currently,  the  Rayleigh  is  the  only  proven  quantitative  model  we  know  of  that 
forecasts  R&D  budget  profiles.  We  define  success  as  being  able  to  improve  our  ability  to 
forecast  completed  budget  profiles  with  the  Weibull  model  verses  the  Rayleigh  model. 
We  compare  the  two  models  by  averaging  the  calculated  correlation  between  actual  and 
forecasted  budget  profiles. 
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We  forecast  budget  profiles  for  the  128  completed  R&D  programs  using  the 


Rayleigh  and  Weibull  models  and  compare  the  calculated  average  correlation  for  each 
model.  Table  23  summaries  our  results  and  highlights  a  significant  improvement  in 
forecasting  budget  profiles  with  the  Weibull  model  as  opposed  to  the  Rayleigh.  On 
average,  we  increase  our  forecasting  ability  60  percent  using  the  Weibull  model  verses 
the  Rayleigh  model.  Our  findings  show  that  the  Weibull  model  is  a  better  tool  to  forecast 
R&D  program  budget  profiles  than  the  current  Rayleigh  model. 


Table  23.  Comparing  Weibull  and  Rayleigh-Based  Budgets 


Correlation  (c) 

Correlation  Distribution 

%  Correlation  Distribution 

Range 

Rayleigh 

Weibull 

Rayleigh 

WeibuU 

c  <  0.0 

67 

5 

52% 

4% 

0.0<c<0.1 

3 

3 

2% 

2% 

0.1  <c<0.2 

10 

3 

8% 

2% 

0.2<c<0.3 

6 

5 

5% 

4% 

0.3  <c<0.4 

6 

8 

5% 

6% 

0.4<c<0.5 

14 

13 

11% 

10% 

0.5  <  c  <  0.6 

5 

12 

4% 

9% 

0.6<c<0.7 

5 

17 

4% 

13% 

0.7<c<0.8 

3 

23 

2% 

18% 

0.8<c<0.9 

3 

19 

2% 

15% 

0.9  <c<  1.0 

6 

20 

5% 

16% 

Total 

128 

128 

100% 

100% 

Summary 

c<.5 

106 

37 

83% 

29% 

c>.5 

22 

91 

17% 

71% 

Correlation  Category 

Weibull 

Rayleigh 

Delta 

Average  Correlation 

0.6068 

0.0021 

0.6047 

Minimum  Correlation 

-0.9984 

-0.9051 

-0.0934 

Maximum  Correlation 

0.9986 

0.9599 

0.0387 
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Our  results  also  show  that  83  percent  of  the  Rayleigh  modeled  budget  profiles  are 
less  than  50  percent  correlated  to  the  actual  budget  profiles.  However,  the  low  average 
correlation  of  0.21  percent  between  the  Rayleigh  and  actual  budget  profiles  is  a  result  of 
52  percent  of  the  Rayleigh  modeled  budget  profiles  being  negatively  correlated  with 
actual  budget  profiles.  Because  the  Rayleigh  model  functions  with  a  fixed  shape 
parameter  of  two,  projected  budget  profiles  are  over  estimate  up  front  and  under 
estimated  in  the  tails  (inversely  estimated)  for  67  of  the  128  programs,  resulting  in 
negative  correlations.  As  an  example,  Figure  22  shows  the  Army  Tactical  Missile 
System  (ATACMS)  completed  R&D  program  budget.  The  correlation  between  the 
ATACMS  budget  profile  and  the  Weibull  and  Rayleigh  modeled  budget  profiles  is 
0.8954  and -0.1 798  respeetively. 


80  81  82  83  84  85  86  87  88  89  90 

Fiscal  Year 


Figure  22.  Rayleigh  and  Weibull  Model  vs.  Actual  ATACMS  Budget  Profile 
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Chapter  Summary 

This  chapter  provides  the  results  of  applying  the  methodology  to  128  completed 
R&D  defense  program  budget  profiles.  The  goodness-of-fit  results  support  our 
assumption  that  the  Weibull  model  fits  R&D  program  expenditures.  We  present  the 
Weibull  shape  and  scale  predictive  models,  and  demonstrate  that  both  models  are 
statistically  significant.  We  validate  the  predictive  regression  models  by  showing  that 
100  and  96  percent  of  the  least  squares  estimated  shape  and  scale  values  fall  within  a  95 
percent  prediction  interval.  Using  the  regression  models  to  predict  shape  and  scale  we 
forecast  Weibull-based  budgets  for  128  completed  R&D  programs  and  report  an  average 
correlation  of  61  percent.  Moreover,  we  define  research  success  as  the  ability  to  forecast 
budget  profiles  at  a  higher  degree  of  aeeuraey  using  the  Weibull  as  opposed  to  the 
Rayleigh  model.  This  ehapter  reports  signifieant  results,  showing  that  the  Weibull  out 
performs  the  Rayleigh  model  in  foreeasting  128  eompleted  R&D  program  budgets  on 
average  60  pereent.  Chapter  Five  summarizes  and  eoncludes  our  thesis  effort,  while 
addressing  possible  limitations  and  future  researeh. 
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V.  Conclusions 


Chapter  Overview 

This  chapter  presents  summary  findings  and  conclusions  of  this  research  effort. 
We  include  a  research  summary  of  Chapters  One,  Two,  and  Three,  and  provide  a 
summary  of  the  results  from  Chapter  Four.  Next,  we  present  the  limitations  to  this 
research  effort  and  provide  recommendations  for  future  research.  Lastly,  comparative 
conclusions  are  drawn  upon  the  results  of  this  research  effort  to  previous  efforts. 

Research  Summary 

Chapter  One  introduces  the  difficultly  in  forecasting  R&D  program  budget 
profiles  that  meet  fiscal  year  expenditure  requirements.  In  addition,  it  relays  that 
expenditure  shortfalls  explain  over  50  percent  of  cost  overruns  and  schedule  slips.  We 
suggest  the  Weibull  in  lieu  of  the  Rayleigh  distribution  to  improve  modeling  of  R&D 
expenditures.  Finally,  the  chapter  concludes  with  the  purpose  of  determining  a 
mathematical  relationship  that  predicts  the  necessary  shape  and  scale  parameters  to 
forecast  Weibull-based  program  budget  profiles. 

Chapter  Two  summarizes  previous  research  in  modeling  R&D  expenditures.  We 
discuss  several  research  efforts  that  support  Rayleigh-based  expenditures  and  budget 
projections.  Flowever,  we  discuss  several  theoretical  limitations  to  the  Rayleigh  and 
introduce  the  Weibull  model  to  mitigate  those  limitations.  We  present  the  shape  and 
scale  parameters  necessary  to  forecast  Weibull-based  budget  profiles.  We  conclude  the 
chapter  with  an  example  of  Lee,  Hogue,  and  Gallagher’s  (1997)  method  of  forecasting 
Rayleigh-based  budget  profiles  given  a  point  estimate. 
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Chapter  Three  contains  our  methodology  to  determine  predictive  relationships  for 
the  Weibull  shape  and  scale  parameters.  We  present  the  criteria  for  data  selection.  We 
explain  the  steps  to  convert  R&D  budgets  to  expenditures  and  our  process  of  nonlinear 
estimation  of  the  Weibull  shape  and  scale  parameters  from  the  converted  expenditures. 
Finally,  we  present  the  steps  to  building  our  predictive  shape  and  scale  models  using 
multiple  regression  techniques. 

Chapter  Four  presents  the  results  of  our  methodology  applied  to  128  completed 
R&D  programs.  Using  goodness-of-fit  statistics  we  demonstrate  that  R&D  expenditures 
are  Weibull  distributed.  Using  102  completed  R&D  defense  programs,  we  develop 
regression  models  to  predict  the  necessary  Weibull  shape  and  scale  parameters.  We  use 
the  remaining  26  completed  R&D  programs  to  validate  the  robustness  of  our  regression 
models  by  showing  that  100  and  96  percent  of  the  least  squares  estimated  shape  and  scale 
values  respectively,  fall  within  our  95  percent  prediction  intervals.  To  determine  the 
Weibull  model’s  budget  projection  capability,  we  compare  128  completed  R&D  program 
budgets  to  Weibull  modeled  budgets  and  report  an  average  correlation  of  0.607.  To 
determine  the  significance  of  our  results  we  compare  the  same  128  completed  program 
budgets  to  Rayleigh  modeled  budgets.  Success  is  defined  as  the  ability  to  improve 
budget  profile  forecasts  with  the  Weibull  as  opposed  to  the  Rayleigh  model.  Using  the 
Weibull  over  the  Rayleigh  model  when  applying  Lee,  Flogue,  and  Gallagher’s  (1997) 
methodology,  we  improve  initial  budget  profile  projections  on  average  60  percent. 
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Limitations 


The  macro  view  of  this  research  effort  seeks  to  minimize  R&D  program  cost  and 
schedule  growth  by  developing  a  better  quantitative  approach  to  forecast  initial  program 
budget  profiles.  However,  several  factors  outside  of  the  scope  of  this  research  contribute 
to  R&D  program  cost  overrun  and  schedule  slip.  Our  focus  narrows  the  research  scope  to 
funding  constraints  attributed  to  budget  profiles  that  do  not  meet  fiscal  expenditure 
requirements.  Within  this  scope  of  focus  we  find  other  limitations  to  forecasting 
Weibull-based  budget  profiles.  First,  R&D  programs  with  four  or  less  budget  years 
rarely  execute  expenditures  that  follow  any  distribution  consistently.  Therefore, 
forecasted  Weibull-based  budgets  on  average  show  poor  correlations  to  final  programs 
with  four  or  less  budget  years.  Secondly,  our  research  only  applies  to  Army,  Navy,  and 
Air  Force  Aequisition  Category  One  (ACAT  I)  R&D  programs.  This  is  due  to  our  data 
source.  The  Department  of  Defense  only  requires  system  program  offices  to  submit  a 
Selected  Acquisition  Report  (SAR)  for  ACAT  I  programs. 

Future  Research 

We  show  in  this  research  that  the  Weibull  out  performs  the  Rayleigh  model  in 
projecting  R&D  program  budget  profiles.  Recommendations  for  future  research  include: 
a  determination  whether  the  Weibull  model  out  performs  current  practices  in  forecasting 
initial  R&D  budgets,  and  applying  the  research  methodology  to  other  existing  data  bases 
that  include  lower  ACAT  R&D  programs.  We  could  not  explore  performance 
comparisons  between  historical  initial  forecasted  R&D  program  budgets  and  Weibull- 
modeled  budgets  based  on  initial  program  cost  and  schedule  estimates  to  the  final 
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program  budgets,  because  only  13  of  the  128  programs  had  initial  budget  profile 
estimates.  This  number  of  programs  is  too  low  to  draw  any  statistical  conclusions. 

Conclusion 

We  conclude  by  stating  that  our  proposed  methodology  provides  the  requisite 
shape  and  scale  regression  models  to  forecast  Weibull-based  R&D  budgets.  Currently, 
the  Rayleigh  model  serves  as  the  only  quantitative  tool  to  project  budget  profiles.  The 
Weibull  out  performs  the  Rayleigh  in  forecasting  128  completed  R&D  budgets  on 
average  60  percent.  We  consider  an  average  increase  of  60  percent  in  budget  profile 
prediction  accuracy  excellent  results.  Thus,  we  achieve  our  research  objective  and 
provide  an  improved  R&D  budget-forecasting  tool. 
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Appendix  A.  Program  Regression  Model  Building  Data 


Table  24,  Program  Model  Building  Data 


#  Programs 

Lead  System 
Service  Type 

Dur  -  Y 

Dur 

Locn 

7 

Scale 

d 

Shape 

P 

1.  A-6E/F 

N 

Aircraft 

5.000 

5 

0.000 

3.811 

2.649 

2.  ABRAMS  UPGRADE 

A 

Vehicle 

10.000 

10 

0.000 

6.501 

2.598 

3.  ACM 

AF 

Missile 

3.582 

5 

1.418 

2.483 

1.755 

4.  ADDS 

A 

Electronic 

8.000 

8 

0.000 

5.683 

2.098 

5.  AFATDS 

A 

Electronic 

15.000 

15 

0.000 

12.314 

3.163 

6.  ALCM 

AF 

Missile 

6.000 

6 

0.000 

4.764 

3.322 

7.  AMRAAM 

AF 

Missile 

11.000 

11 

0.000 

8.341 

3.754 

8.  AN/BSY-1 

N 

Electronic 

9.000 

9 

0.000 

4.501 

1.776 

9.  AN/BSY-2 

N 

Electronic 

16.000 

16 

0.000 

11.376 

4.025 

10.  AN/SQQ-89 

N 

Electronic 

8.000 

8 

0.000 

6.450 

2.929 

11.  AN/TTC-39 

A 

Electronic 

9.000 

9 

0.000 

6.419 

2.423 

12.  AGE  6 

N 

Ship 

7.000 

7 

0.000 

4.050 

2.436 

13.  APACHE  (AH-64) 

A 

Aircraft 

10.000 

10 

0.000 

7.194 

2.665 

14.  ASAS 

A 

Electronic 

8.000 

8 

0.000 

5.776 

2.590 

15.  AS  AT 

AF 

Missile 

13.000 

13 

0.000 

11.660 

5.487 

16.  ASM  LOS  AT 

A 

Vehiele 

8.000 

8 

0.000 

6.671 

3.718 

17.  ASPJ 

N 

Eleetronie 

11.002 

12 

0.998 

6.361 

2.154 

18.  ATACMS 

A 

Missile 

11.000 

11 

0.000 

8.432 

3.512 

19.  ATACMS-APAM 

A 

Missile 

4.000 

4 

0.000 

2.521 

1.931 

20.  ATARS 

AF 

Eleetronie 

6.914 

9 

2.086 

4.308 

2.022 

21.  AV-8B 

N 

Aircraft 

8.000 

8 

0.000 

6.201 

3.233 

22.  AVENGER 

A 

Vehicle 

3.000 

3 

0.000 

2.546 

1.957 

23.  B-1  CMUP-JDAM 

AF 

Eleetronie 

3.941 

5 

1.059 

2.926 

2.070 

24.  B-IB 

AF 

Aircraft 

4.000 

4 

0.000 

3.471 

2.622 

25.  B-2 

AF 

Aircraft 

11.520 

13 

1.480 

7.323 

2.191 

26.  B-52  OAS/CMI 

AF 

Electronic 

4.980 

5 

0.020 

4.068 

2.735 

27.  BATTLESHIP 

N 

Ship 

6.000 

6 

0.000 

4.017 

1.930 

28.  BFVS 

A 

Vehicle 

15.000 

15 

0.000 

12.631 

4.498 

29.  BFVS  UPGRADE 

A 

Vehicle 

7.000 

7 

0.000 

4.221 

2.010 

30.  BLACKHAWK 

A 

Aircraft 

6.818 

9 

2.182 

5.675 

3.421 

31.  C/MH-53E 

N 

Aircraft 

5.967 

6 

0.033 

3.930 

1.915 

32.  C-17A 

AF 

Aircraft 

9.000 

9 

0.000 

8.366 

5.922 

33.  CAPTOR 

N 

Munition 

12.000 

12 

0.000 

9.420 

4.063 

34.  CH-47D 

A 

Aircraft 

4.937 

5 

0.063 

3.440 

2.041 
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#  Programs 

Lead  System 
Service  Type 

Dur  -  Y 

Dur 

Locn 

r 

Scale 

5 

Shape 

P 

35.  CH-60S 

N 

Aircraft 

4.955 

5 

0.045 

3.690 

2.610 

36.  CMU 

AF 

Electronic  22.000 

22 

0.000 

14.976 

3.458 

37.  COPPERHEAD 

A 

Munition 

9.000 

9 

0.000 

7.319 

3.366 

38.  CSRL 

AF 

Munition 

4.000 

4 

0.000 

3.298 

2.390 

39.  CV  HELO 

N 

Aircraft 

4.000 

4 

0.000 

2.512 

1.676 

40.  CVN  68 

N 

Ship 

4.730 

5 

0.270 

3.467 

2.383 

41.  DDG-51 

N 

Ship 

7.000 

7 

0.000 

5.389 

2.863 

42.  DSCS  III 

AF 

Space 

5.396 

6 

0.604 

3.400 

1.771 

43.  E-3A 

AF 

Aircraft 

11.000 

11 

0.000 

9.701 

4.993 

44.  E-6A 

N 

Electronic 

5.247 

6 

0.753 

4.568 

3.289 

45.  EF-lllA 

AF 

Aircraft 

6.913 

7 

0.087 

4.772 

2.606 

46.  EJS 

AF 

Electronic 

6.000 

6 

0.000 

5.190 

3.324 

47.  F/A-18 

N 

Aircraft 

5.095 

6 

0.905 

3.768 

2.549 

48.  F/A-18  E/F 

N 

Aircraft 

8.308 

9 

0.692 

3.783 

1.642 

49.  F-14 

N 

Aircraft 

3.706 

4 

0.294 

2.572 

1.871 

50.  F-14D 

N 

Aircraft 

4.621 

6 

1.379 

3.306 

2.182 

51.  F-16 

AF 

Aircraft 

2.489 

3 

0.511 

2.381 

2.457 

52.  FAAD  C2I 

A 

Electronic 

8.000 

8 

0.000 

6.213 

2.302 

53.  FAADS  NLOS 

A 

Missile 

5.000 

5 

0.000 

3.452 

2.332 

54.  FDS 

N 

Electronic 

13.000 

13 

0.000 

9.444 

3.408 

55.  FFG  7 

N 

Ship 

2.156 

3 

0.844 

1.367 

1.405 

56.  FMTV 

A 

Vehicle 

3.690 

4 

0.310 

2.515 

1.855 

57.  GLCM 

AF 

Missile 

6.000 

6 

0.000 

4.428 

2.678 

58.  HARM 

N 

Missile 

10.000 

10 

0.000 

8.386 

3.266 

59.  HARPOON 

N 

Missile 

6.000 

6 

0.000 

4.970 

3.312 

60.  HH-60D 

AF 

Aircraft 

5.000 

5 

0.000 

3.619 

2.521 

61.  I-S/AAMPE 

AF 

Electronic 

3.318 

4 

0.682 

1.632 

1.125 

62.  lUS 

AF 

Space 

7.247 

8 

0.753 

5.196 

2.196 

63.  JAVELIN 

A 

Missile 

12.000 

12 

0.000 

6.583 

2.273 

64.  JDAM 

AF 

Munition 

7.879 

8 

0.121 

4.401 

1.891 

65.  JSIPS 

AF 

Electronic 

8.000 

8 

0.000 

5.383 

2.372 

66.  JSOW 

N 

Missile 

15.000 

15 

0.000 

9.642 

3.911 

67.  JSTARS 

AF 

Electronic 

15.000 

15 

0.000 

10.756 

2.693 

68.  JTUAV 

N 

Aircraft 

7.000 

7 

0.000 

6.263 

3.237 

69.  KC-135R 

AF 

Aircraft 

6.000 

6 

0.000 

5.416 

3.396 

70.  KIOWA 

A 

Aircraft 

6.000 

6 

0.000 

4.457 

2.914 

71.  LAMPS 

N 

Electronic 

13.000 

13 

0.000 

11.186 

4.574 
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#  Programs 

Lead  System 
Service  Type 

Dur  -  Y 

Dur 

Locn 

r 

Scale 

5 

Shape 

P 

72.  LANTIRN 

AF 

Electronic 

7.000 

7 

0.000 

5.631 

2.764 

73.  LASER  HELLFIRE 

A 

Missile 

11.000 

11 

0.000 

8.553 

3.769 

74.  LBG 

AF 

Munition 

5.000 

5 

0.000 

3.532 

2.129 

75.  EC  AC 

N 

Ship 

5.000 

5 

0.000 

4.392 

3.889 

76.  LHD  1 

N 

Ship 

3.321 

5 

1.679 

1.275 

1.014 

77.  LONGBOW  APACHE 

A 

Electronic 

11.000 

11 

0.000 

7.405 

2.296 

78.  LONGBOW  HELLFIRE 

A 

Missile 

5.000 

5 

0.000 

3.429 

2.022 

79.  LSD  41 

N 

Ship 

5.000 

5 

0.000 

3.509 

2.033 

80.  LSD  41  CV 

N 

Ship 

6.000 

6 

0.000 

4.940 

3.514 

81.  MAVERICK 

AF 

Missile 

8.000 

8 

0.000 

6.357 

3.975 

82.  MCM  1 

N 

Ship 

5.000 

5 

0.000 

3.769 

2.031 

83.  MCS 

A 

Electronic 

4.000 

4 

0.000 

3.239 

2.205 

84.  MHC  51 

N 

Ship 

3.546 

4 

0.454 

2.520 

1.815 

85.  MK  48  ADCAP 

N 

Missile 

7.000 

7 

0.000 

5.484 

2.894 

86.  MK  50  TORPEDO 

N 

Missile 

15.000 

15 

0.000 

11.668 

3.148 

87.  MLRS 

A 

Munition 

3.947 

5 

1.053 

3.335 

2.799 

88.  MLRS-TGW 

A 

Munition 

13.000 

13 

0.000 

10.673 

3.634 

89.  MLS 

AF 

Electronic 

7.000 

7 

0.000 

6.077 

4.176 

90.  MMlll  GRP 

AF 

Missile 

7.000 

7 

0.000 

4.571 

2.207 

91.  MMIII  PRP 

AF 

Missile 

7.000 

7 

0.000 

5.172 

2.614 

92.  NAS 

AF 

Electronic 

9.000 

9 

0.000 

6.557 

2.330 

93.  NAVSTAR  GPS 

AF 

Electronic 

13.000 

13 

0.000 

9.900 

2.429 

94.  NESP 

N 

Electronic 

12.000 

12 

0.000 

7.554 

1.816 

95.  OTH-B 

AF 

Electronic 

2.698 

3 

0.302 

2.550 

2.487 

96.  PATRIOT 

A 

Missile 

16.000 

16 

0.000 

11.990 

3.005 

97.  PATRIOT(Fire  Unit) 

A 

Missile 

10.000 

10 

0.000 

7.015 

2.327 

98.  PATRIOT(Missile  Seg) 

N 

Missile 

19.000 

19 

0.000 

14.683 

3.845 

99.  PEACEKEEPER 

AF 

Missile 

4.000 

4 

0.000 

2.644 

1.768 

100.  PERSHING  II 

A 

Missile 

8.000 

8 

0.000 

6.985 

3.739 

101.  PHOENIX 

N 

Missile 

8.000 

8 

0.000 

5.227 

2.565 

102.  PLS 

A 

Vehicle 

2.295 

3 

0.705 

1.453 

1.560 

103.  PLSS 

AF 

Electronic 

13.000 

13 

0.000 

11.017 

4.094 

104.  ROTHR 

N 

Electronic 

6.569 

7 

0.431 

4.337 

2.371 

105.  RPV 

A 

Aircraft 

14.000 

14 

0.000 

11.186 

4.043 

106.  RSIP 

AF 

Electronic 

7.787 

8 

0.213 

4.540 

1.694 

107.  SADARM 

A 

Munition 

10.000 

10 

0.000 

7.296 

2.514 

108.  SCAMP 

A 

Space 

4.000 

4 

0.000 

3.499 

2.775 
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#  Programs 

Lead  System 
Service  Type 

Dur  -  Y 

Dur 

Locn 

r 

Scale 

5 

Shape 

P 

109.  SFW 

AF 

Munition 

10.000 

10 

0.000 

6.615 

2.202 

no.  SGT  YORK  GUN 

A 

Munition 

5.821 

6 

0.179 

4.637 

2.627 

111.  SH-60R 

N 

Aircraft 

14.000 

14 

0.000 

9.365 

2.620 

112.  SINCGARS 

A 

Electronic 

7.426 

8 

0.574 

5.573 

2.967 

113.  SLAT 

N 

Missile 

12.000 

12 

0.000 

8.832 

2.379 

114.  SPARROW 

N 

Missile 

6.000 

6 

0.000 

4.593 

2.127 

115.  SRAM  II 

AF 

Missile 

8.000 

8 

0.000 

6.866 

4.123 

116.  SSN21 

N 

Ship 

8.000 

8 

0.000 

7.002 

4.310 

117.  STINGER 

A 

Missile 

7.000 

7 

0.000 

5.367 

2.684 

118.  T-45TS 

N 

Aircraft 

10.666 

13 

2.334 

7.428 

2.917 

119.  T-46A 

AF 

Aircraft 

3.056 

5 

1.944 

3.059 

3.185 

120.  TACTAS 

N 

Electronic 

9.000 

9 

0.000 

6.024 

2.389 

121.  T-AGOS 

N 

Ship 

5.000 

5 

0.000 

3.678 

1.894 

122.  TOMAHAWK 

N 

Missile 

8.997 

10 

1.003 

5.694 

1.916 

123.  TOW  2 

A 

Missile 

4.000 

4 

0.000 

3.419 

2.822 

124.  TRIDENT  II  MSL 

N 

Missile 

7.780 

10 

2.220 

6.689 

4.314 

125.  TRI-TAC 

AF 

Eleetronie 

8.395 

10 

1.605 

5.391 

2.043 

126.  UGM-84 

N 

Missile 

4.789 

5 

0.211 

2.404 

1.765 

127.  V-22 

N 

Aireraft 

19.000 

19 

0.000 

13.165 

2.379 

128.  WISWAM 

AF 

Eleetronie 

11.278 

12 

0.722 

7.270 

2.152 
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Appendix  B.  Service  Raw  Inflation  Indices 


Table  25.  RDT&E  FY2000  Raw  Inflation  Indices 


FY 

Air  Force 

Army 

msm 

FY 

Air  Force 

1965 

0.2100 

0.2123 

0.2163 

1993 

0.9011 

0.9038 

0.9038 

1966 

0.2156 

0.2194 

0.2221 

1994 

0.9192 

0.9219 

0.9219 

1967 

0.2225 

0.2273 

0.2293 

1995 

0.9366 

0.9394 

0.9394 

1968 

0.2305 

0.2350 

0.2376 

1996 

0.9554 

0.9582 

0.9582 

1969 

0.2414 

0.2424 

0.2488 

1997 

0.9754 

0.9754 

0.9754 

1970 

0.2546 

0.2577 

0.2625 

1998 

0.9822 

0.9822 

0.9822 

1971 

0.2676 

0.2693 

0.2760 

1999 

0.9901 

0.9901 

0.9901 

1972 

0.2799 

0.2796 

0.2887 

2000 

1.0000 

1.0000 

1.0000 

1973 

0.2923 

0.2910 

0.3012 

2001 

1.0150 

1.0150 

1.0150 

1974 

0.3151 

0.3082 

0.3253 

2002 

1.0302 

1.0302 

1.0302 

1975 

0.3491 

0.3547 

0.3608 

2003 

1.0457 

1.0457 

1.0457 

1976 

0.3732 

0.3750 

0.3847 

2004 

1.0666 

1.0666 

1.0666 

1977 

0.3985 

0.4057 

0.4060 

2005 

1.0879 

1.0879 

1.0879 

1978 

0.4256 

0.4346 

0.4336 

2006 

1.1097 

1.1097 

1.1097 

1979 

0.4614 

0.4741 

0.4700 

2007 

1.1319 

1.1319 

1.1319 

1980 

0.5048 

0.5243 

0.5198 

2008 

1.1545 

1.1545 

1.1545 

1981 

0.5648 

0.5799 

0.5749 

2009 

1.1776 

1.1776 

1.1776 

1982 

0.6168 

0.6240 

0.6186 

2010 

1.2012 

1.2012 

1.2012 

1983 

0.6470 

0.6489 

0.6489 

2011 

1.2252 

1.2252 

1.2252 

1984 

0.6716 

0.6736 

0.6736 

2012 

1.2497 

1.2497 

1.2497 

1985 

0.6944 

0.6964 

0.6965 

2013 

1.2747 

1.2747 

1.2747 

1986 

0.7139 

0.7160 

0.7160 

2014 

1.3002 

1.3002 

1.3002 

1987 

0.7332 

0.7353 

0.7353 

2015 

1.3262 

1.3262 

1.3262 

1988 

0.7552 

0.7574 

0.7574 

2016 

1.3527 

1.3527 

1.3527 

1989 

0.7869 

0.7892 

0.7892 

2017 

1.3798 

1.3798 

1.3798 

1990 

0.8184 

0.8216 

0.8208 

2018 

1.4073 

1.4073 

1.4073 

1991 

0.8535 

0.8569 

0.8561 

2019 

1.4355 

1.4355 

1.4355 

1992 

0.8774 

0.8826 

0.8800 
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